[00:02] *** ndiddy has joined #archiveteam-bs [00:11] *** wp494 has quit IRC (Read error: Operation timed out) [00:12] *** wp494 has joined #archiveteam-bs [00:22] *** Oddly has quit IRC (Ping timeout: 255 seconds) [00:34] *** BlueMax has joined #archiveteam-bs [01:02] *** step has quit IRC (ZNC 1.7.1 - https://znc.in) [01:13] *** hdch has quit IRC (Ping timeout: 265 seconds) [01:14] *** step has joined #archiveteam-bs [01:17] *** tomaspark has quit IRC (Ping timeout: 255 seconds) [01:19] *** Swiss- has joined #archiveteam-bs [01:24] *** tomaspark has joined #archiveteam-bs [01:28] *** hdch has joined #archiveteam-bs [01:53] *** Sanqui has quit IRC (Remote host closed the connection) [02:00] *** m007a83 has joined #archiveteam-bs [02:32] *** hdch has quit IRC (Ping timeout: 265 seconds) [02:46] arkiver: Could you set up a target soon please? We need to get this started, only have roughly a day left anyway (or maybe a little more depending on when they're shutting it down, but there will be a massive number of job failures). [02:47] Otherwise I suggest we just continue with the current target and sort out the mess afterwards. Should be easy enough to distinguish between the old and new files based on file modification date. [02:52] *** Sanqui has joined #archiveteam-bs [02:52] *** svchfoo3 sets mode: +o Sanqui [02:57] Pinging some more people involved with targets: SketchCow HCross Kaz ^ [03:18] JAA: DId someone request a rsync target? [03:19] kiska: That's also an option I think. [03:19] *** exoire has quit IRC (Read error: Operation timed out) [03:20] The situation is that there's data from the old scripts on FOS, so continuing there would be messy. A fresh target would work. [03:26] JAA: What is the item on IA? [03:28] JAA kiska speaking of rsync targets, would either of you be able to help with the newsgrabber rsync master being out of disk space or know who to ping about that? [03:28] HCross: ^ [03:30] kiska: I don't think there is an item/collection yet. [03:30] I see [03:30] http://fos.textfiles.com/pipeline.html [03:31] Mover script not running, and anyway only very little data so far because most thread IDs below 1 million don't exist. [03:39] *** MR9K has quit IRC (Quit: tilde lounge - https://irc.tilde.team) [04:06] I'll leave rsync config until I have had breakfast... I borked up config [04:07] kiska it is 3:07 pm [04:07] Why have you not had breakfast [04:08] That's not that bad. I had breakfast at around 19:00 today/yesterday. [04:10] Actually no, later than that. Anyway, that's -ot material. [04:10] *** hdch has joined #archiveteam-bs [04:22] JAA: How much space do you think I will need for the rsync target for the forums? [04:22] kiska: No idea yet. [04:23] We went through almost 50k thread IDs with only 4 GB of data, but that doesn't help at all because almost none of those threads exist. I intend to restart from ID 1 million, the threshold above which most threads appear to be. [04:24] Uh no, 50k items with 10 thread IDs each, so 500k thread IDs. [04:24] There are ~4.5 million IDs total. [04:32] *** qw3rty119 has joined #archiveteam-bs [04:36] *** qw3rty118 has quit IRC (Ping timeout: 600 seconds) [04:43] *** odemg has quit IRC (Ping timeout: 265 seconds) [04:55] *** odemg has joined #archiveteam-bs [05:04] *** hdch has quit IRC (Remote host closed the connection) [05:15] JAA: I got the rsync target up on lightsail. rsync://rsync.kiska.pw/uol/:downloader/ [05:15] kiska: On Lightsail? How much capacity does that have? [05:18] The $80/mo plan [05:19] 16GB ram, 4 vCPU, 320GB ssd, 6TB transfer l [05:20] *** coldon2dr has quit IRC (Quit: https://quassel-irc.org - Chat comfortably. Anywhere.) [05:20] Alright, we'll see how it goes. Thanks. [05:21] right [05:21] I'll start setting up the megawarc factory and associated services [05:21] still need a target? [05:21] oh i see [05:25] Adding 100k items from 1 million up now. [05:27] And we're live. There will be many errors. [05:28] JAA: Can you login to the rsync target? username is ubuntu [05:29] kiska: Yup [05:29] Can we keep window 0 for the megawarc processes? [05:31] I don't mind how it's set up. I also have zero experience with targets and the megawarc factory. [05:31] Neither do I xD [05:33] So it looks like we're grabbing something. [05:33] Very slow though. [05:34] I did limit it to only 50 items per minute. On the other hand, it doesn't seem like the rate has anything to do with the errors. [05:34] 10 200s then sleeping [05:35] I got a lot of errors while testing as well. Perhaps other people are trying to download the forums as well. [05:35] Does this project have its own channel yet? [05:35] Nope, we didn't come up with a good name yet I think. [05:36] maybe #UhOhL [05:36] A channel for all of UOL, that is. The forums are just one part. [05:36] Flashfire: lol [05:36] Oh, I like that. [05:36] is #UhOhL good [05:36] I may not be able to code but stupid puns are my specialty [05:39] *** Albardin has joined #archiveteam-bs [05:39] *** w0rmybak has joined #archiveteam-bs [05:39] *** kiskabak has joined #archiveteam-bs [05:44] SketchCow: When do I get a collection that I can upload UOL stuff into? [06:29] *** m007a83_ has joined #archiveteam-bs [06:32] *** m007a83 has quit IRC (Ping timeout: 252 seconds) [06:32] *** Swiss- has quit IRC (Read error: Connection reset by peer) [06:33] *** Swiss- has joined #archiveteam-bs [06:55] *** MR9K has joined #archiveteam-bs [07:19] https://web.archive.org/save/ now offers saving of passworded stuff cool [07:31] *** Swiss- has quit IRC (Read error: Connection reset by peer) [07:48] *** ndiddy has quit IRC () [07:52] *** MR9K has quit IRC (Quit: tilde lounge - https://irc.tilde.team) [07:55] *** MR9K has joined #archiveteam-bs [08:07] *** m007a83_ is now known as m007a83 [08:13] *** MR9K has quit IRC (Quit: tilde lounge - https://irc.tilde.team) [08:13] *** MR9K has joined #archiveteam-bs [08:40] *** fredgido_ has joined #archiveteam-bs [08:47] *** fredgido has quit IRC (Read error: Operation timed out) [08:49] *** m007a83 has quit IRC (Read error: Connection reset by peer) [09:08] *** wp494 has quit IRC (Ping timeout: 265 seconds) [09:09] *** wp494 has joined #archiveteam-bs [09:27] *** m007a83 has joined #archiveteam-bs [09:28] *** Oddly has joined #archiveteam-bs [09:54] *** hook54321 has quit IRC (Quit: Connection closed for inactivity) [10:12] *** fredgido has joined #archiveteam-bs [10:13] *** fredgido_ has quit IRC (Ping timeout: 252 seconds) [11:42] *** hdch has joined #archiveteam-bs [11:54] *** BlueMax has quit IRC (Quit: Leaving) [12:09] *** Oddly has quit IRC (Read error: Operation timed out) [12:10] *** Oddly has joined #archiveteam-bs [13:49] *** Sian1468 has quit IRC () [13:52] *** hdch has quit IRC (Quit: oops) [15:00] *** Mateon1 has quit IRC (Ping timeout: 360 seconds) [15:00] *** Mateon1 has joined #archiveteam-bs [16:24] *** hook54321 has joined #archiveteam-bs [16:25] *** Odd0002_ has joined #archiveteam-bs [16:26] *** Odd0002 has quit IRC (Ping timeout: 252 seconds) [16:26] *** Odd0002_ is now known as Odd0002 [16:30] *** Oddly has quit IRC (Read error: Operation timed out) [16:39] *** exoire has joined #archiveteam-bs [16:40] *** exoire has quit IRC (Client Quit) [16:45] *** exoire has joined #archiveteam-bs [16:49] XIVDB is offline. My grab didn't finish in time, but I can't tell you right now how much it managed to grab. [16:50] *** Terry has joined #archiveteam-bs [17:59] *** Terry is now known as NotGLaDOS [18:10] *** wp494 has quit IRC (Read error: Operation timed out) [18:10] *** wp494 has joined #archiveteam-bs [19:13] *** daan101 has joined #archiveteam-bs [19:21] *** Oddly has joined #archiveteam-bs [19:50] *** hook54321 has quit IRC () [19:50] *** hook54321 has joined #archiveteam-bs [21:15] *** daan101 has quit IRC (Leaving) [21:26] *** ppsym has quit IRC (Remote host closed the connection) [21:31] *** brayden has quit IRC (Ping timeout: 260 seconds) [21:47] *** brayden has joined #archiveteam-bs [21:47] *** swebb sets mode: +o brayden [21:48] *** exoire has quit IRC (Read error: Operation timed out) [22:35] *** tomaspark has quit IRC (Read error: Connection reset by peer) [22:48] JAA: I see you fixed UOL :) [22:49] *** SimpBrain has joined #archiveteam-bs [22:51] *** BlueMax has joined #archiveteam-bs [23:08] *** Oddly has quit IRC (Ping timeout: 255 seconds)