[00:06] *** Stiletti has quit IRC (Read error: Operation timed out) [00:06] *** Stiletti has joined #archiveteam [00:16] Ah, trickle seems quite useful [00:18] *** Swizzle has joined #archiveteam [00:20] *** Aranje has quit IRC (Quit: Three sheets to the wind) [00:23] Guess I just wait for a response on key then before firing up :) [00:52] *** pizzaiolo has quit IRC (Quit: pizzaiolo) [01:09] *** Stiletti has quit IRC (Read error: Operation timed out) [01:09] *** Stiletti has joined #archiveteam [01:22] *** drumstick has quit IRC (Read error: Operation timed out) [01:27] *** nertzy has joined #archiveteam [01:46] *** Stiletti has quit IRC (Read error: Operation timed out) [01:46] *** Stiletti has joined #archiveteam [01:46] *** schbirid2 has joined #archiveteam [01:49] *** schbirid has quit IRC (Read error: Operation timed out) [02:09] *** jrwr has quit IRC (Read error: Operation timed out) [02:10] *** jrwr has joined #archiveteam [02:15] *** matthusby has quit IRC (Remote host closed the connection) [02:33] *** matthusby has joined #archiveteam [02:34] *** Stiletti has quit IRC (Read error: Operation timed out) [02:34] *** Stiletti has joined #archiveteam [02:43] *** drumstick has joined #archiveteam [03:00] *** kitties has quit IRC (Quit: Connection closed for inactivity) [03:09] *** Stiletti has quit IRC (Read error: Operation timed out) [03:09] *** Stiletti has joined #archiveteam [03:44] *** qw3rty15 has joined #archiveteam [03:50] *** qw3rty14 has quit IRC (Read error: Operation timed out) [04:10] *** BubuAnabe has quit IRC (Ping timeout: 268 seconds) [04:19] *** Mateon1 has quit IRC (Ping timeout: 268 seconds) [04:19] *** Mateon1 has joined #archiveteam [04:19] *** Stiletti has quit IRC (Read error: Operation timed out) [04:20] *** Stiletti has joined #archiveteam [04:21] *** matthusb_ has joined #archiveteam [04:21] *** matthusby has quit IRC (Read error: Operation timed out) [04:21] *** matthusb_ has quit IRC (Remote host closed the connection) [04:21] *** matthusby has joined #archiveteam [04:34] *** Sk1d has quit IRC (Ping timeout: 194 seconds) [04:40] *** Sk1d has joined #archiveteam [04:57] *** BubuAnabe has joined #archiveteam [05:20] *** mmm has joined #archiveteam [05:22] *** toohighto has joined #archiveteam [05:42] *** Stiletti has quit IRC (Read error: Operation timed out) [05:42] *** Stiletti has joined #archiveteam [06:50] Wow, 60GB for ArchiveBot? So it doesn't just download small pieces and immediately upload them again? [06:51] *** Stiletti has quit IRC (Read error: Operation timed out) [06:51] *** Stiletti has joined #archiveteam [06:52] Also, do you know a Digital Ocean block storage volume would work? [06:52] I've been looking for a way to help out [06:52] *** Swizzle has quit IRC (Read error: Operation timed out) [06:53] *** matthusby has quit IRC (Remote host closed the connection) [06:53] *** mmm has quit IRC (Quit: Page closed) [07:05] yipdw: Are you actually accepting new archivebot pipelines? Our wiki says no: http://archiveteam.org/index.php?title=ArchiveBot#Volunteer_a_Node [07:05] that is still correct [07:05] the support overhead is too high [07:06] qwerty0: we do download in 5 GB pieces and upload them again. the problem is when you have many processes downloading many small pieces [07:06] log files are also not chunked [07:06] ah [07:06] but you can limit the number of processes? [07:07] yes, most people don't [07:07] and you still have the log file issue [07:08] if you end up with a site that has a gazillion plus one URLs your log file will be quite large [07:08] well, if DO block storage works, i think it's fine [07:08] it's not too expensive for 60GB [07:08] and in any case, the bigger issue is if you're not actually accepting new pipelines [07:09] or is what Asparagir was asking was for just donations of resources for existing pipeline operators? [07:09] I don't know what Asparagir asked for [07:10] "ust a general note: ArchiveBot needs more volunteers to set up and run pipelines. We only have a few people running a few pipelines, and some of them have bugs and need restarting or are almost full, and things like that. We need more capacity. [07:10] So if you have free credits at AWS or Digital Ocean or whatever, or are willing to pay for a small server (a 60 GB hard drive should do, don't need much computing power), this is your time to step up." [07:10] I stopped adding keys for new people because each new person adds support overhead that I do not have the time to manage [07:11] i.e. things like "my pipeline went offline", "please remove the key", etc [07:11] right, sure, makes sense [07:11] those are manual tasks [07:11] there are some people who have access to the control system, they're also quite busy [07:11] so you don't know what Asparagir is planning or talking about? [07:11] just trying to determine whether I can help out and who to talk to [07:12] I can examine the logs to get a better idea, I can also ask her [07:13] if the registration/deregistration system was fully automated then all of the aforementioned overhead would go away. unfortunately it's a long way from that and I haven't yet gotten to the point where I can bump that to priority 1 [07:13] right, so at the moment all new operators have to go through you? [07:14] or someone who has root access to the control node [07:14] okay, so there's others? and Asparagir is one? [07:14] that set is me, FalconK, Sanqui [07:15] I don't see Asparagir's SSH key in the list, I can add it if she wants me to [07:15] okay cool, just getting it straight [07:32] *** Stiletti has quit IRC (Read error: Operation timed out) [07:32] *** Stiletti has joined #archiveteam [07:37] Im ready to send my key to whoever [08:04] *** Stiletti has quit IRC (Read error: Operation timed out) [08:04] *** Stiletti has joined #archiveteam [08:11] *** HarryCros has quit IRC (Read error: Connection reset by peer) [08:18] *** HarryCros has joined #archiveteam [08:43] *** Stiletti has quit IRC (Read error: Operation timed out) [08:43] *** Stiletti has joined #archiveteam [08:46] *** drumstick has quit IRC (Read error: Operation timed out) [08:50] *** drumstick has joined #archiveteam [09:05] What happened with http://archiveteam.org/index.php?title=PDF_2016 ? [09:08] *** Honno has joined #archiveteam [09:19] *** Honno_ has quit IRC (Read error: Operation timed out) [09:27] http://archiveteam.org/index.php?title=PDF_manuals [09:39] So who adds most of the things in the ArchiveBot queue? Looks like it's usually really busy [09:41] people in #archivebot [09:48] right, of course [09:54] *ba dum tss* [10:10] qwerty0: One thing that yipdw failed to mention by the way: wpull's database can also grow to massive sizes for large jobs (think millions of URLs). You can also run into problems when there are very large files in jobs; we tried to grab betaarchive a while ago, for example, and that downloaded several preview images of Windows 10 in parallel, crashing the pipeline. [10:10] haha oops yeah that makes sense [10:11] just smashes through some assumptions one might make when designing ArchiveBot [10:23] Asparagir: is it still worth me keeping my pipeline up? [10:27] HCross2: What hosting are you using? [10:28] M247, Vienna [10:29] oh, hadn't heard of them [11:00] *** drumstick has quit IRC (Read error: Operation timed out) [11:04] *** Stiletti has quit IRC (Read error: Operation timed out) [11:05] *** Stiletti has joined #archiveteam [11:21] *** j08nY has joined #archiveteam [11:22] *** bRick5772 has joined #archiveteam [11:50] Starting archivebot manually seemed to barbaric, so I made a thing: https://github.com/valgrind/abot-scripts [11:50] Untested, because I don't have an account to test with yet... [11:50] *** fie has quit IRC (Read error: Operation timed out) [11:50] And wrong channel... [11:58] *** BartoCH has quit IRC (Ping timeout: 260 seconds) [12:02] *** pizzaiolo has joined #archiveteam [12:06] *** BartoCH has joined #archiveteam [12:32] *** Stiletti has quit IRC (Read error: Operation timed out) [12:32] *** Stiletti has joined #archiveteam [12:44] at SHA I was unable to find arkiver or joepie91, so I did a lightningtalk about Archive Team a bit unprepared [12:44] I hope I did it justice [12:44] that was really at the edge of my comfort zone, so much fun [12:56] atluxity: talks are good! my general rule is "i should get better at talks, so don't say no when people say i should give one" [12:57] :) [13:22] *** Stiletti has quit IRC (Read error: Operation timed out) [13:22] *** Stiletti has joined #archiveteam [13:24] *** matthusby has joined #archiveteam [13:40] *** matthusby has quit IRC (Remote host closed the connection) [13:47] *** marvinw is now known as ivan [14:07] *** BubuAnabe has quit IRC (Ping timeout: 268 seconds) [14:18] *** cadbury_ has joined #archiveteam [14:41] *** Stiletti has quit IRC (Read error: Operation timed out) [15:27] *** alex___ has joined #archiveteam [16:05] *** matthusby has joined #archiveteam [16:14] *** matthusb_ has joined #archiveteam [16:14] *** matthusby has quit IRC (Read error: Connection reset by peer) [16:20] *** matthusb_ has quit IRC (Remote host closed the connection) [16:31] *** alex___ has quit IRC (Quit: take care ye all. Have fun!) [16:32] *** dashcloud has quit IRC (Remote host closed the connection) [16:35] *** matthusby has joined #archiveteam [16:38] *** dashcloud has joined #archiveteam [16:39] *** Aranje has joined #archiveteam [16:40] *** toohighto has quit IRC (Remote host closed the connection) [16:53] atluxity: oops, pinged you in #archiveteam-bs , but looks like you aren't there [16:53] where you at? [16:53] (please join #archiveteam-bs ) [16:53] :) [17:29] *** Aranje has quit IRC (Ping timeout: 245 seconds) [17:38] *** BubuAnabe has joined #archiveteam [17:46] *** matthusby has quit IRC (Remote host closed the connection) [18:20] *** trvz has quit IRC (Quit: ZNC 1.6.5+deb1 - http://znc.in) [18:35] *** fie has joined #archiveteam [18:50] *** wabu has quit IRC (Read error: Operation timed out) [19:08] *** matthusby has joined #archiveteam [19:08] *** matthusby has quit IRC (Remote host closed the connection) [19:10] *** matthusby has joined #archiveteam [19:12] *** matthusb_ has joined #archiveteam [19:12] *** matthusby has quit IRC (Read error: Connection reset by peer) [19:16] *** wabu has joined #archiveteam [20:09] *** toohighto has joined #archiveteam [20:11] *** kitties has joined #archiveteam [20:16] *** Swizzle has joined #archiveteam [20:26] *** Swizzle has quit IRC (Quit: Leaving) [20:29] *** Lord_Nigh has quit IRC (Read error: Operation timed out) [20:46] *** Aranje has joined #archiveteam [20:48] *** Lord_Nigh has joined #archiveteam [20:52] *** TC01 has joined #archiveteam [21:01] *** HarryCros has quit IRC (Read error: Connection reset by peer) [21:02] *** HarryCros has joined #archiveteam [21:27] *** bwn has quit IRC (Ping timeout: 268 seconds) [21:28] *** bRick5772 has quit IRC (Quit: Leaving.) [21:31] *** bwn has joined #archiveteam [21:36] *** Aranje has quit IRC (Quit: Three sheets to the wind) [21:45] *** Odd0002 has quit IRC (Remote host closed the connection) [22:03] *** matthusb_ has quit IRC (Read error: Operation timed out) [22:34] *** drumstick has joined #archiveteam [22:39] *** matthusby has joined #archiveteam [22:43] *** William has joined #archiveteam [22:50] *** JerryStie has quit IRC (Read error: Operation timed out) [22:54] *** ZexaronS has quit IRC (Quit: Leaving) [23:20] *** username1 has joined #archiveteam [23:23] *** schbirid2 has quit IRC (Read error: Operation timed out) [23:36] *** schbirid2 has joined #archiveteam [23:39] *** username1 has quit IRC (Read error: Operation timed out) [23:39] *** William has quit IRC () [23:46] *** username1 has joined #archiveteam [23:46] *** kristian_ has joined #archiveteam [23:49] *** schbirid2 has quit IRC (Read error: Operation timed out) [23:49] *** matthusby has quit IRC (Remote host closed the connection) [23:54] *** schbirid2 has joined #archiveteam [23:55] *** nmjhyu has joined #archiveteam [23:55] *** nmjhyu has quit IRC (Client Quit) [23:55] *** username1 has quit IRC (Read error: Operation timed out)