[00:03] That's good to know. My home-made script, even if it worked perfectly, was only going to archive the top 1000 lists, plus a list of threads that I personally found interesting. [00:03] *** BlueMax has joined #archiveteam-bs [00:04] The latter are few enough in number that I can probably still do them with my script. [00:06] I'll just have to archive them less frequently - I don't think Reddit likes it when the WB machine hits them once every 10 seconds, if that's even possible. [00:06] Yeah, and there are many people trying to archive Reddit pages through the WBM as well. [00:07] I recall TwitchLeecher recently stopped working on subscriber-only VODs. I believe it said something about Twitch changing the API to prevent it. I downloaded a few with the VideoDownloadHelper Firefox extension. There's probably an easier way, but weren't really enough to videos to look for it. [00:07] Oh really? And there was me thinking the constant 429 errors were my fault! [00:08] Oh and Ryz, I never get close to the 12-15 pages/min limit, so no need to log in. [00:10] Yeah, the WBM backend only has so many IPs, and your save requests get mixed with everyone else's. [00:14] *** britmob has quit IRC (Read error: Connection reset by peer) [00:14] *** britmob has joined #archiveteam-bs [00:15] Then I'm guessing the sudden uptick in 429 errors was from someone else deploying a script to archive Reddit, probably more aggressively than mine. [00:16] *** benjinsmi has joined #archiveteam-bs [00:17] Which seems more than likely, considering it was done on the last night before the Monday Banwave. [00:19] *** benjins has quit IRC (Ping timeout: 265 seconds) [00:26] *** OrIdow6 has quit IRC (Ping timeout: 265 seconds) [00:31] Do you think it's worth archiving some of these comment threads again but with ?limit=500 added to the end of the URL? [00:31] Obviously for the larger threads only. That way we get 500 comments instead of 200. [00:34] How many threads are we talking about here? [00:38] At a guess, maybe 2000 or so per subreddit? How many subs has ArchiveBot crawled so far? [00:41] Quick log scan says around 1k https://old.reddit.com/r/* jobs since 2018 (which is when "old" started to be a thing). [00:44] This is perhaps a bigger undertaking than I imagined. [00:45] Well, Reddit has roughly 1 billion threads. [00:46] *** OrIdow6 has joined #archiveteam-bs [00:46] It doesn't make sense to do large-scale archiving of it through ArchiveBot. [00:46] That's why I mentioned #shreddit earlier. [00:47] Very true. Either way, some of the at-risk subs produce a couple of dozen new threads per day, so about a 1k per month, which makes archiving them a higher priority. [00:48] I'll remind myself to issue the request to AB in a couple of weeks then [00:50] So how would I do it when the time comes - ask someone in the #archivebot channel? Would it help if I came with a pre-prepared list of URLs? [00:56] I'm conscious to avoid archiving duplicates. [00:56] *** britmob has quit IRC (Read error: Connection reset by peer) [00:56] *** britmob has joined #archiveteam-bs [00:57] Yeah, #archivebot would be best. Ping me (i.e. mention my nick) so it doesn't get drowned in the bot noise there. [00:57] And yes, a list would be ideal. [01:01] *** pew has quit IRC (Ping timeout: 265 seconds) [01:02] Thanks JAA. I'll have my script scrape the Reddit API then check to see where the archived URLs stop, but not archive them myself, then upload the rest of a Pastebin document or something. [01:08] Sounds good. [01:10] There are some even smaller tasks I think my script is better equipped to handle, like daily snapshots of subreddit front pages, Twitter tags, etc. I'll now be able to focus more on that. [01:12] Anyway, it's getting late. But thanks for your help with this. Goodnight! [01:13] *** pew has joined #archiveteam-bs [01:18] Repeated archival of certain things is something various people here have mentioned in the past, but we don't have anything for it really. [01:18] Happy to help, good night. [01:20] JAA: I'm thinking that the current script that I have could be run every year or so (along with the dedup) – once the unarchived tweets per gov account reaches a certain threshold [01:21] Idk really – just something that I was considering if this turns out well [01:21] maxfan8: Sure, but this is something different. I mean things like "grab https://example.org/ every 72 hours". [01:21] Ah ok [01:27] *** atphoenix has quit IRC (Read error: Connection reset by peer) [01:27] *** atphoenix has joined #archiveteam-bs [01:48] *** notroot has joined #archiveteam-bs [02:13] *** HP_Archiv has joined #archiveteam-bs [02:21] *** HP_Archiv has quit IRC (Quit: Leaving) [03:03] *** DogsRNice has quit IRC (Read error: Connection reset by peer) [03:20] *** qw3rty_ has joined #archiveteam-bs [03:27] *** qw3rty has quit IRC (Read error: Operation timed out) [03:30] *** atphoenix has quit IRC (Ping timeout: 265 seconds) [03:30] *** atphoenix has joined #archiveteam-bs [04:31] *** britmob has quit IRC (Read error: Connection reset by peer) [04:32] *** britmob has joined #archiveteam-bs [04:33] *** benjinss has joined #archiveteam-bs [04:35] *** benjinsmi has quit IRC (Ping timeout: 265 seconds) [05:49] *** Lord_Nigh has quit IRC (Quit: ZNC - http://znc.in) [05:51] *** Lord_Nigh has joined #archiveteam-bs [06:01] *** kyledrake has quit IRC (hub.efnet.us irc.Prison.NET) [06:01] *** ndiddy has quit IRC (hub.efnet.us irc.Prison.NET) [06:01] *** sivoais has quit IRC (hub.efnet.us irc.Prison.NET) [06:01] *** Pixi` has quit IRC (hub.efnet.us irc.Prison.NET) [06:01] *** scorche has quit IRC (hub.efnet.us irc.Prison.NET) [06:01] *** achip has quit IRC (hub.efnet.us irc.Prison.NET) [06:01] *** Somebody2 has quit IRC (hub.efnet.us irc.Prison.NET) [06:01] *** riking_ has quit IRC (Read error: Connection reset by peer) [06:01] *** DrasticAc has quit IRC (Read error: Connection reset by peer) [06:01] *** pnJay has quit IRC (Read error: Connection reset by peer) [06:01] *** diggan has quit IRC (Read error: Connection reset by peer) [06:01] *** horkermon has quit IRC (Read error: Connection reset by peer) [06:01] *** maxfan8_ has joined #archiveteam-bs [06:03] *** Maylay has quit IRC (Read error: Operation timed out) [06:03] *** Maylay has joined #archiveteam-bs [06:03] *** diggan has joined #archiveteam-bs [06:03] *** DrasticAc has joined #archiveteam-bs [06:03] *** pnJay has joined #archiveteam-bs [06:03] *** horkermon has joined #archiveteam-bs [06:03] *** riking_ has joined #archiveteam-bs [06:04] *** maxfan8_ has quit IRC (Client Quit) [06:04] *** maxfan8_ has joined #archiveteam-bs [06:05] *** maxfan8_ has quit IRC (Client Quit) [06:06] *** maxfan8_ has joined #archiveteam-bs [06:06] *** maxfan8 has quit IRC (Ping timeout: 610 seconds) [06:06] *** maxfan8_ has quit IRC (Client Quit) [06:06] *** maxfan8 has joined #archiveteam-bs [06:17] *** Raccoon has joined #archiveteam-bs [06:18] *** kyledrake has joined #archiveteam-bs [06:18] *** ndiddy has joined #archiveteam-bs [06:18] *** sivoais has joined #archiveteam-bs [06:18] *** Pixi` has joined #archiveteam-bs [06:18] *** scorche has joined #archiveteam-bs [06:18] *** achip has joined #archiveteam-bs [06:18] *** Somebody2 has joined #archiveteam-bs [06:25] *** ranma_ has joined #archiveteam-bs [06:25] *** Stilett0 has joined #archiveteam-bs [06:27] *** prq has quit IRC (Read error: Operation timed out) [06:27] *** superkuh has quit IRC (Write error: Broken pipe) [06:27] *** underscor has quit IRC (Write error: Broken pipe) [06:27] *** robogoat has quit IRC (Write error: Broken pipe) [06:27] *** underscor has joined #archiveteam-bs [06:27] *** mtntmnky_ has quit IRC (Read error: Operation timed out) [06:27] *** nyany has quit IRC (Read error: Operation timed out) [06:28] *** jrwr has quit IRC (Ping timeout: 260 seconds) [06:28] *** SynMonger has quit IRC (Read error: Operation timed out) [06:28] *** SynMonger has joined #archiveteam-bs [06:29] *** DigiDigi has quit IRC (Read error: Operation timed out) [06:29] *** phirephl- has quit IRC (Ping timeout: 376 seconds) [06:29] *** Stiletto has quit IRC (Ping timeout: 376 seconds) [06:30] *** pie_[bnc] has quit IRC (Read error: Operation timed out) [06:30] *** svchfoo1 has quit IRC (Read error: Operation timed out) [06:30] *** Igloo has quit IRC (Read error: Operation timed out) [06:30] *** ranma has quit IRC (Write error: Broken pipe) [06:30] *** Larsenv has quit IRC (Read error: Operation timed out) [06:30] *** sknebel has quit IRC (Read error: Operation timed out) [06:30] *** pie_ has joined #archiveteam-bs [06:30] *** sknebel has joined #archiveteam-bs [06:30] *** Darkstar has quit IRC (Read error: Operation timed out) [06:31] *** superkuh has joined #archiveteam-bs [06:31] *** chfoo has quit IRC (Read error: Operation timed out) [06:31] *** Iglooop1 has quit IRC (Read error: Operation timed out) [06:31] *** atphoenix has quit IRC (Read error: Operation timed out) [06:31] *** chfoo has joined #archiveteam-bs [06:31] *** godane has quit IRC (Read error: Operation timed out) [06:32] *** atphoenix has joined #archiveteam-bs [06:32] *** robogoat has joined #archiveteam-bs [06:33] *** yano_ has joined #archiveteam-bs [06:34] *** phirephly has joined #archiveteam-bs [06:35] *** godane has joined #archiveteam-bs [06:35] *** Larsenv has joined #archiveteam-bs [06:37] *** Frogging has quit IRC (Ping timeout: 298 seconds) [06:37] *** simon816 has quit IRC (Ping timeout: 298 seconds) [06:37] *** jodizzle_ has joined #archiveteam-bs [06:39] *** jodizzle has quit IRC (Ping timeout: 312 seconds) [06:39] *** _niklas_ has quit IRC (Ping timeout: 312 seconds) [06:39] *** pikami_ has joined #archiveteam-bs [06:39] *** Selavii has joined #archiveteam-bs [06:39] *** SynMonger has quit IRC (Read error: Connection reset by peer) [06:39] *** Darkstar has joined #archiveteam-bs [06:39] *** Frogging has joined #archiveteam-bs [06:39] *** JAA_ has joined #archiveteam-bs [06:39] *** AlsoJAA sets mode: +o JAA_ [06:39] *** yano has quit IRC (Read error: Connection reset by peer) [06:39] *** kiska has quit IRC (Ping timeout: 257 seconds) [06:39] *** cm has quit IRC (Ping timeout: 257 seconds) [06:39] *** pikami has quit IRC (Ping timeout: 257 seconds) [06:39] *** swebb has quit IRC (Ping timeout: 257 seconds) [06:39] *** bleb has joined #archiveteam-bs [06:39] *** simon816 has joined #archiveteam-bs [06:39] *** bleb is now known as cm [06:39] *** kiska has joined #archiveteam-bs [06:40] *** JAA has quit IRC (Read error: Connection reset by peer) [06:40] *** Selavi has quit IRC (Read error: Connection reset by peer) [06:40] *** Selavii is now known as Selavi [06:41] *** swebb has joined #archiveteam-bs [06:41] *** SynMonger has joined #archiveteam-bs [06:41] *** _niklas has joined #archiveteam-bs [06:46] *** nyany has joined #archiveteam-bs [06:46] *** DigiDigi has joined #archiveteam-bs [06:46] *** prq has joined #archiveteam-bs [06:46] *** Iglooop1 has joined #archiveteam-bs [06:47] *** svchfoo1 has joined #archiveteam-bs [06:47] *** Craigle has quit IRC (Quit: The Lounge - https://thelounge.chat) [06:47] *** svchfoo3 sets mode: +o svchfoo1 [06:48] *** jrwr has joined #archiveteam-bs [06:49] *** Craigle has joined #archiveteam-bs [06:51] *** Igloo has joined #archiveteam-bs [06:54] *** mtntmnky_ has joined #archiveteam-bs [08:10] *** BlueMaxim has joined #archiveteam-bs [08:10] *** BlueMax has quit IRC (Read error: Connection reset by peer) [08:10] *** DLoader_ has joined #archiveteam-bs [08:12] *** Selavii has joined #archiveteam-bs [08:13] *** jmtd has joined #archiveteam-bs [08:14] *** ablabiX has joined #archiveteam-bs [08:15] *** thejsa_ has joined #archiveteam-bs [08:16] *** closure_ has joined #archiveteam-bs [08:16] *** sirvy_ has joined #archiveteam-bs [08:19] *** BartoCH has quit IRC (Ping timeout: 615 seconds) [08:19] *** sirvy has quit IRC (Ping timeout: 615 seconds) [08:19] *** amelia386 has quit IRC (Read error: Connection timed out) [08:19] *** picklefac has quit IRC (Read error: Connection timed out) [08:19] *** Vito` has quit IRC (Read error: Connection timed out) [08:20] *** Selavi has quit IRC (Ping timeout: 615 seconds) [08:20] *** c0mpass has quit IRC (Ping timeout: 1229 seconds) [08:20] *** tchaypo_ has quit IRC (Ping timeout: 1229 seconds) [08:20] *** Selavii is now known as Selavi [08:20] *** BartoCH has joined #archiveteam-bs [08:20] *** DrasticAc has quit IRC (Ping timeout: 1230 seconds) [08:20] *** Kaz has quit IRC (Ping timeout: 1230 seconds) [08:20] *** lenary has quit IRC (Ping timeout: 1230 seconds) [08:20] *** closure has quit IRC (Ping timeout: 745 seconds) [08:21] *** horkermon has quit IRC (Ping timeout: 1230 seconds) [08:21] *** diggan has quit IRC (Ping timeout: 1230 seconds) [08:21] *** pnJay has quit IRC (Ping timeout: 1230 seconds) [08:21] *** Xibalba has quit IRC (Ping timeout: 745 seconds) [08:21] *** DLoader has quit IRC (Ping timeout: 745 seconds) [08:21] *** ablabiX is now known as Xibalba [08:21] *** Jon| has quit IRC (Ping timeout: 745 seconds) [08:21] *** DLoader_ is now known as DLoader [08:22] *** Vito` has joined #archiveteam-bs [08:22] *** starlord has quit IRC (Ping timeout: 1230 seconds) [08:22] *** thejsa has quit IRC (Ping timeout: 745 seconds) [08:23] *** justcool3 has quit IRC (Ping timeout: 1230 seconds) [08:24] *** Ctrl-S___ has quit IRC (Ping timeout: 1230 seconds) [08:24] *** mattl has quit IRC (Ping timeout: 1230 seconds) [08:24] *** fallenoak has quit IRC (Ping timeout: 1230 seconds) [08:25] *** picklefac has joined #archiveteam-bs [08:25] *** HCross has quit IRC (Ping timeout: 1230 seconds) [08:25] *** jesse-s has quit IRC (Ping timeout: 1230 seconds) [08:26] *** riking_ has quit IRC (Read error: Connection timed out) [08:27] *** JSharp___ has quit IRC (Read error: Connection timed out) [08:27] *** ThisAsYou has quit IRC (Read error: Connection timed out) [08:27] *** hook54321 has quit IRC (Read error: Connection timed out) [08:27] *** revi has quit IRC (Read error: Connection timed out) [08:27] *** abartov__ has quit IRC (Read error: Connection timed out) [08:30] *** Kaz has joined #archiveteam-bs [08:33] *** starlord has joined #archiveteam-bs [08:38] *** amelia386 has joined #archiveteam-bs [08:42] *** pnJay has joined #archiveteam-bs [08:44] *** pnJay has quit IRC (Read error: Connection reset by peer) [08:44] *** phirephl- has joined #archiveteam-bs [08:45] *** phirephly has quit IRC (Read error: Operation timed out) [08:46] *** c0mpass has joined #archiveteam-bs [08:46] *** fallenoak has joined #archiveteam-bs [08:46] *** jesse-s has joined #archiveteam-bs [08:46] *** abartov__ has joined #archiveteam-bs [08:46] *** jesse-s has quit IRC (Read error: Connection reset by peer) [08:46] *** fallenoak has quit IRC (Read error: Connection reset by peer) [08:46] *** c0mpass has quit IRC (Read error: Connection reset by peer) [08:46] *** HCross has joined #archiveteam-bs [08:48] *** Pixi has joined #archiveteam-bs [08:49] *** SynMonger has quit IRC (hub.efnet.us irc.Prison.NET) [08:49] *** superkuh has quit IRC (hub.efnet.us irc.Prison.NET) [08:49] *** kyledrake has quit IRC (hub.efnet.us irc.Prison.NET) [08:49] *** ndiddy has quit IRC (hub.efnet.us irc.Prison.NET) [08:49] *** sivoais has quit IRC (hub.efnet.us irc.Prison.NET) [08:49] *** Pixi` has quit IRC (hub.efnet.us irc.Prison.NET) [08:49] *** scorche has quit IRC (hub.efnet.us irc.Prison.NET) [08:49] *** achip has quit IRC (hub.efnet.us irc.Prison.NET) [08:49] *** Somebody2 has quit IRC (hub.efnet.us irc.Prison.NET) [08:49] *** amelia386 has quit IRC (Read error: Connection reset by peer) [08:49] *** starlord has quit IRC (Read error: Connection reset by peer) [08:49] *** picklefac has quit IRC (Read error: Connection reset by peer) [08:49] *** Vito` has quit IRC (Read error: Connection reset by peer) [08:49] *** abartov__ has quit IRC (Read error: Connection reset by peer) [08:49] *** HCross has quit IRC (Write error: Connection reset by peer) [08:55] *** riking_ has joined #archiveteam-bs [08:56] *** riking_ has quit IRC (Read error: Connection reset by peer) [08:57] *** ThisAsYou has joined #archiveteam-bs [08:58] *** ThisAsYou has quit IRC (Read error: Connection reset by peer) [08:58] *** sivoais_ has joined #archiveteam-bs [09:02] *** hook54321 has joined #archiveteam-bs [09:02] *** jesse-s has joined #archiveteam-bs [09:03] *** c0mpass has joined #archiveteam-bs [09:03] *** fallenoak has joined #archiveteam-bs [09:07] *** mattl has joined #archiveteam-bs [09:07] *** justcool3 has joined #archiveteam-bs [09:07] *** Ctrl-S___ has joined #archiveteam-bs [09:07] *** tchaypo_ has joined #archiveteam-bs [09:08] *** HCross has joined #archiveteam-bs [09:09] *** tchaypo_ has quit IRC (Read error: Connection reset by peer) [09:09] *** fallenoak has quit IRC (Read error: Connection reset by peer) [09:09] *** c0mpass has quit IRC (Read error: Connection reset by peer) [09:09] *** jesse-s has quit IRC (Read error: Connection reset by peer) [09:09] *** HCross has quit IRC (Read error: Connection reset by peer) [09:10] *** hook54321 has quit IRC () [09:10] *** ThisAsYou has joined #archiveteam-bs [09:17] *** revi has joined #archiveteam-bs [09:17] *** ThisAsYou has quit IRC (Read error: Connection reset by peer) [09:21] *** hook54321 has joined #archiveteam-bs [09:22] *** HCross has joined #archiveteam-bs [09:22] *** horkermon has joined #archiveteam-bs [09:23] *** diggan has joined #archiveteam-bs [09:23] *** pnJay has joined #archiveteam-bs [09:24] *** ThisAsYou has joined #archiveteam-bs [09:26] *** pnJay has quit IRC (Read error: Connection reset by peer) [09:26] *** diggan has quit IRC (Read error: Connection reset by peer) [09:26] *** horkermon has quit IRC (Read error: Connection reset by peer) [09:27] *** amelia386 has joined #archiveteam-bs [09:28] *** picklefac has joined #archiveteam-bs [09:28] *** ThisAsYou has quit IRC (Read error: Connection reset by peer) [09:28] *** fallenoak has joined #archiveteam-bs [09:28] *** fallenoak has quit IRC (Read error: Connection reset by peer) [09:29] *** lenary has joined #archiveteam-bs [09:30] *** HCross has quit IRC (Read error: Connection reset by peer) [09:31] *** hook54321 has quit IRC () [09:33] *** hook54321 has joined #archiveteam-bs [09:33] *** hook54321 has quit IRC (Client Quit) [09:33] *** c0mpass has joined #archiveteam-bs [09:34] *** ThisAsYou has joined #archiveteam-bs [09:35] *** fallenoak has joined #archiveteam-bs [09:36] *** HCross has joined #archiveteam-bs [09:37] *** HCross has quit IRC (Read error: Connection reset by peer) [09:37] *** riking_ has joined #archiveteam-bs [09:38] *** ThisAsYou has quit IRC (Read error: Connection reset by peer) [09:39] *** tchaypo_ has joined #archiveteam-bs [09:39] *** tchaypo_ has quit IRC (Read error: Connection reset by peer) [09:39] *** fallenoak has quit IRC (Read error: Connection reset by peer) [09:39] *** c0mpass has quit IRC (Read error: Connection reset by peer) [09:42] *** SynMonger has joined #archiveteam-bs [09:42] *** superkuh has joined #archiveteam-bs [09:42] *** kyledrake has joined #archiveteam-bs [09:42] *** scorche has joined #archiveteam-bs [09:42] *** achip has joined #archiveteam-bs [09:42] *** Somebody2 has joined #archiveteam-bs [09:42] *** mattl has quit IRC (Read error: Connection reset by peer) [09:42] *** justcool3 has quit IRC (Read error: Connection reset by peer) [09:42] *** picklefac has quit IRC (Read error: Connection reset by peer) [09:42] *** amelia386 has quit IRC (Read error: Connection reset by peer) [09:42] *** revi has quit IRC (Read error: Connection reset by peer) [09:42] *** Ctrl-S___ has quit IRC (Write error: Connection reset by peer) [09:43] *** abartov__ has joined #archiveteam-bs [09:45] *** tchaypo_ has joined #archiveteam-bs [09:45] *** horkermon has joined #archiveteam-bs [09:45] *** c0mpass has joined #archiveteam-bs [09:46] *** c0mpass has quit IRC (Read error: Connection reset by peer) [09:46] *** HCross has joined #archiveteam-bs [09:46] *** pnJay has joined #archiveteam-bs [09:46] *** pnJay has quit IRC (Read error: Connection reset by peer) [09:46] *** horkermon has quit IRC (Read error: Connection reset by peer) [09:46] *** riking_ has quit IRC (Read error: Connection reset by peer) [09:46] *** lenary has quit IRC (Read error: Connection timed out) [09:52] *** jesse-s has joined #archiveteam-bs [09:52] *** jesse-s has quit IRC (Read error: Connection reset by peer) [09:52] *** tchaypo_ has quit IRC (Read error: Connection reset by peer) [09:53] *** riking_ has joined #archiveteam-bs [09:54] *** pnJay has joined #archiveteam-bs [09:54] *** pnJay has quit IRC (Read error: Connection reset by peer) [09:57] *** tchaypo_ has joined #archiveteam-bs [09:58] *** c0mpass has joined #archiveteam-bs [09:59] *** jesse-s has joined #archiveteam-bs [10:00] *** jesse-s has quit IRC (Read error: Connection reset by peer) [10:00] *** c0mpass has quit IRC (Read error: Connection reset by peer) [10:00] *** tchaypo_ has quit IRC (Read error: Connection reset by peer) [10:00] *** horkermon has joined #archiveteam-bs [10:01] *** pnJay has joined #archiveteam-bs [10:07] *** pnJay has quit IRC (Read error: Connection reset by peer) [10:07] *** horkermon has quit IRC (Read error: Connection reset by peer) [10:07] *** riking_ has quit IRC (Read error: Connection reset by peer) [10:12] *** tchaypo_ has joined #archiveteam-bs [10:13] *** tchaypo_ has quit IRC (Read error: Connection reset by peer) [10:19] *** c0mpass has joined #archiveteam-bs [10:20] *** tchaypo_ has joined #archiveteam-bs [10:20] *** ThisAsYou has joined #archiveteam-bs [10:20] *** jesse-s has joined #archiveteam-bs [10:21] *** jesse-s has quit IRC (Read error: Connection reset by peer) [10:21] *** c0mpass has quit IRC (Read error: Connection reset by peer) [10:27] *** riking_ has joined #archiveteam-bs [10:27] *** riking_ has quit IRC (Read error: Connection reset by peer) [10:28] *** JSharp___ has joined #archiveteam-bs [10:32] *** JSharp___ has quit IRC (Read error: Connection reset by peer) [10:35] *** riking_ has joined #archiveteam-bs [10:36] *** fallenoak has joined #archiveteam-bs [10:37] *** jesse-s has joined #archiveteam-bs [10:37] *** pnJay has joined #archiveteam-bs [10:38] *** tchaypo_ has quit IRC (Read error: Connection timed out) [10:38] *** horkermon has joined #archiveteam-bs [10:39] *** tchaypo_ has joined #archiveteam-bs [10:41] *** c0mpass has joined #archiveteam-bs [11:00] *** JSharp___ has joined #archiveteam-bs [11:11] *** c0mpass has quit IRC (Read error: Connection reset by peer) [11:12] *** tchaypo_ has quit IRC (Read error: Connection reset by peer) [11:12] *** jesse-s has quit IRC (Read error: Connection reset by peer) [11:12] *** fallenoak has quit IRC (Read error: Connection reset by peer) [11:20] *** horkermon has quit IRC (Read error: Connection reset by peer) [11:20] *** pnJay has quit IRC (Read error: Connection reset by peer) [11:20] *** riking_ has quit IRC (Read error: Connection reset by peer) [11:22] *** jesse-s has joined #archiveteam-bs [11:31] *** tchaypo_ has joined #archiveteam-bs [11:31] *** tchaypo_ has quit IRC (Read error: Connection reset by peer) [11:31] *** jesse-s has quit IRC (Read error: Connection reset by peer) [11:38] *** jesse-s has joined #archiveteam-bs [11:38] *** tchaypo_ has joined #archiveteam-bs [11:40] *** horkermon has joined #archiveteam-bs [11:40] *** pnJay has joined #archiveteam-bs [11:40] *** riking_ has joined #archiveteam-bs [11:40] *** riking_ has quit IRC (Read error: Connection reset by peer) [11:40] *** pnJay has quit IRC (Read error: Connection reset by peer) [11:40] *** horkermon has quit IRC (Read error: Connection reset by peer) [11:42] *** c0mpass has joined #archiveteam-bs [11:42] *** c0mpass has quit IRC (Read error: Connection reset by peer) [11:42] *** jesse-s has quit IRC (Read error: Connection reset by peer) [11:45] *** revi has joined #archiveteam-bs [11:46] *** BlueMaxim has quit IRC (Read error: Connection reset by peer) [11:48] *** starlord has joined #archiveteam-bs [11:52] *** pnJay has joined #archiveteam-bs [11:52] *** tchaypo_ has quit IRC (Read error: Connection reset by peer) [11:52] *** horkermon has joined #archiveteam-bs [11:52] *** riking_ has joined #archiveteam-bs [11:57] *** mattl has joined #archiveteam-bs [11:59] *** c0mpass has joined #archiveteam-bs [11:59] *** tchaypo_ has joined #archiveteam-bs [12:00] *** mattl has quit IRC (Read error: Connection reset by peer) [12:02] *** jesse-s has joined #archiveteam-bs [12:03] *** tchaypo_ has quit IRC (Read error: Connection reset by peer) [12:03] *** c0mpass has quit IRC (Read error: Connection reset by peer) [12:05] *** mattl has joined #archiveteam-bs [12:14] *** jesse-s has quit IRC (Read error: Connection reset by peer) [12:15] *** mattl has quit IRC (Read error: Connection reset by peer) [12:20] *** jesse-s has joined #archiveteam-bs [12:23] *** c0mpass has joined #archiveteam-bs [12:23] *** tchaypo_ has joined #archiveteam-bs [12:28] *** DrasticAc has joined #archiveteam-bs [12:30] *** diggan has joined #archiveteam-bs [12:44] *** mattl has joined #archiveteam-bs [13:16] *** fallenoak has joined #archiveteam-bs [13:44] *** amelia386 has joined #archiveteam-bs [13:46] *** mattl has quit IRC (Read error: Connection reset by peer) [14:05] *** mattl has joined #archiveteam-bs [14:05] *** mattl has quit IRC (Read error: Connection reset by peer) [14:07] *** JAA_ is now known as JAA [14:12] *** mattl has joined #archiveteam-bs [14:16] *** mattl has quit IRC (Read error: Connection reset by peer) [14:36] *** revi has quit IRC () [14:38] *** revi has joined #archiveteam-bs [14:42] *** Vito` has joined #archiveteam-bs [14:59] *** mattl has joined #archiveteam-bs [15:06] *** mattl has quit IRC (Read error: Connection reset by peer) [15:21] *** HP_Archiv has joined #archiveteam-bs [15:27] *** ThisAsYou has quit IRC (Read error: Connection reset by peer) [15:33] *** Arcorann_ has quit IRC (Read error: Connection reset by peer) [15:36] *** mattl has joined #archiveteam-bs [15:36] *** mattl has quit IRC (Read error: Connection reset by peer) [15:42] *** HP_Archiv has quit IRC (Quit: Leaving) [15:47] *** ThisAsYou has joined #archiveteam-bs [15:47] *** justcool3 has joined #archiveteam-bs [15:49] *** justcool3 has quit IRC (Read error: Connection reset by peer) [16:00] *** mattl has joined #archiveteam-bs [16:00] *** justcool3 has joined #archiveteam-bs [16:02] *** Ctrl-S___ has joined #archiveteam-bs [16:09] *** mattl has quit IRC (Read error: Connection reset by peer) [16:13] *** ThisAsYou has quit IRC (Read error: Connection reset by peer) [16:19] *** ThisAsYou has joined #archiveteam-bs [16:20] *** Ctrl-S___ has quit IRC (Read error: Connection reset by peer) [16:20] *** justcool3 has quit IRC (Read error: Connection reset by peer) [16:26] *** justcool3 has joined #archiveteam-bs [16:26] *** Ctrl-S___ has joined #archiveteam-bs [16:28] *** mattl has joined #archiveteam-bs [16:29] *** mattl has quit IRC (Read error: Connection reset by peer) [16:29] *** Ctrl-S___ has quit IRC (Write error: Connection reset by peer) [16:29] *** justcool3 has quit IRC (Read error: Connection reset by peer) [16:30] *** schbirid has joined #archiveteam-bs [16:36] *** larryv has joined #archiveteam-bs [16:43] *** girst has quit IRC (Quit: ZNC 1.8.0 - https://znc.in) [16:45] *** girst has joined #archiveteam-bs [16:47] *** mattl has joined #archiveteam-bs [16:47] *** mattl has quit IRC (Read error: Connection reset by peer) [16:49] *** Jon| has joined #archiveteam-bs [16:49] *** jmtd has quit IRC (Ping timeout: 272 seconds) [16:59] *** justcool3 has joined #archiveteam-bs [17:00] *** mattl has joined #archiveteam-bs [17:00] *** Ctrl-S___ has joined #archiveteam-bs [17:08] *** ThisAsYou has quit IRC (Read error: Connection reset by peer) [17:22] *** schbirid has quit IRC (Quit: Leaving) [17:23] *** larryv has quit IRC (Quit: larryv) [17:23] *** Mateon1 has quit IRC (Ping timeout: 745 seconds) [17:24] *** Mateon1 has joined #archiveteam-bs [17:24] *** HP_Archiv has joined #archiveteam-bs [17:38] *** ThisAsYou has joined #archiveteam-bs [17:52] *** mattl has quit IRC (Read error: Connection reset by peer) [17:52] *** justcool3 has quit IRC (Read error: Connection reset by peer) [17:52] *** JSharp___ has quit IRC (Read error: Connection reset by peer) [17:52] *** Vito` has quit IRC (Read error: Connection reset by peer) [17:52] *** amelia386 has quit IRC (Read error: Connection reset by peer) [17:52] *** starlord has quit IRC (Read error: Connection reset by peer) [17:52] *** Ctrl-S___ has quit IRC (Write error: Connection reset by peer) [17:52] *** fallenoak has quit IRC (Write error: Connection reset by peer) [17:52] *** jesse-s has quit IRC (Write error: Connection reset by peer) [17:52] *** pnJay has quit IRC (Write error: Connection reset by peer) [17:52] *** diggan has quit IRC (Write error: Connection reset by peer) [17:52] *** DrasticAc has quit IRC (Write error: Connection reset by peer) [17:52] *** horkermon has quit IRC (Write error: Connection reset by peer) [17:52] *** tchaypo_ has quit IRC (Write error: Connection reset by peer) [17:52] *** c0mpass has quit IRC (Write error: Connection reset by peer) [17:52] *** riking_ has quit IRC (Write error: Connection reset by peer) [17:52] *** abartov__ has quit IRC (Write error: Connection reset by peer) [17:52] *** ThisAsYou has quit IRC (Read error: Connection reset by peer) [17:53] *** HCross has quit IRC (Write error: Connection reset by peer) [17:54] *** Sokar has quit IRC (Ping timeout: 745 seconds) [18:03] *** ThisAsYou has joined #archiveteam-bs [18:04] *** hook54321 has joined #archiveteam-bs [18:05] *** Ctrl-S___ has joined #archiveteam-bs [18:12] *** mattl has joined #archiveteam-bs [18:13] *** mattl has quit IRC (Read error: Connection reset by peer) [18:13] *** hook54321 has quit IRC (Read error: Connection reset by peer) [18:13] *** Ctrl-S___ has quit IRC (Write error: Connection reset by peer) [18:23] *** ThisAsYou has quit IRC (Read error: Connection reset by peer) [18:23] *** horkermon has joined #archiveteam-bs [18:23] *** horkermon has quit IRC (Read error: Connection reset by peer) [18:24] *** HP_Archiv has quit IRC (Quit: Leaving) [18:29] *** ThisAsYou has joined #archiveteam-bs [18:29] *** HCross has joined #archiveteam-bs [18:30] *** horkermon has joined #archiveteam-bs [18:34] *** horkermon has quit IRC (Read error: Connection reset by peer) [18:34] *** HCross has quit IRC (Read error: Connection reset by peer) [18:34] *** ThisAsYou has quit IRC (Read error: Connection reset by peer) [18:35] *** mattl has joined #archiveteam-bs [18:39] *** horkermon has joined #archiveteam-bs [18:44] *** Ctrl-S___ has joined #archiveteam-bs [18:44] *** horkermon has quit IRC (Read error: Connection reset by peer) [18:49] *** horkermon has joined #archiveteam-bs [18:53] *** HCross has joined #archiveteam-bs [18:54] *** ThisAsYou has joined #archiveteam-bs [18:55] *** HCross has quit IRC (Read error: Connection reset by peer) [18:55] *** ThisAsYou has quit IRC (Read error: Connection reset by peer) [19:07] *** ThisAsYou has joined #archiveteam-bs [19:07] *** HCross has joined #archiveteam-bs [19:22] *** xit has joined #archiveteam-bs [19:33] *** xit has quit IRC (Read error: Connection reset by peer) [19:33] *** horkermon has quit IRC (Read error: Connection reset by peer) [19:37] I conclude getting 'all' clips from a twitch account is impossible [19:39] *** horkermon has joined #archiveteam-bs [19:40] *** xit has joined #archiveteam-bs [19:40] the API returns around 1000 results; I tried getting clips for the entire history of the stream, and whenever the API returned more than 200, I split the date interval into two halves and requested them separately, I saw it keep subdividing up to 3-hour intervals [19:41] now I tried a different starting interval which *still* covers the whole history, but when subdividing would cause different intervals that overlap my previous attempt... and it keeps finding new clips, some of them with hundreds of views [19:42] it feels like any change I make keeps digging more data, and I never know if I got all [20:04] *** logchfoo4 starts logging #archiveteam-bs at Thu Jul 09 20:04:14 2020 [20:04] *** logchfoo4 has joined #archiveteam-bs [20:06] *** starlord has joined #archiveteam-bs [20:07] *** ThisAsYou has joined #archiveteam-bs [20:08] *** horkermon has joined #archiveteam-bs [20:15] *** horkermon has quit IRC (Read error: Connection reset by peer) [20:22] *** horkermon has joined #archiveteam-bs [20:26] *** wyatt8750 is now known as wyatt8740 [20:26] *** ThisAsYou has quit IRC (Read error: Connection reset by peer) [20:26] *** xit has joined #archiveteam-bs [20:31] *** ThisAsYou has joined #archiveteam-bs [20:36] *** ThisAsYou has quit IRC (Read error: Connection reset by peer) [20:39] *** HCross has joined #archiveteam-bs [20:39] *** schbirid has joined #archiveteam-bs [20:39] *** HCross has quit IRC (Read error: Connection reset by peer) [20:43] *** ThisAsYou has joined #archiveteam-bs [20:43] *** hook54321 has joined #archiveteam-bs [20:48] *** hook54321 has quit IRC (Read error: Connection reset by peer) [20:52] *** HCross has joined #archiveteam-bs [20:53] *** sHATNER has joined #archiveteam-bs [20:53] *** BlueMax has joined #archiveteam-bs [20:53] *** JSharp___ has joined #archiveteam-bs [20:54] *** Laverne has joined #archiveteam-bs [20:54] *** Meli has joined #archiveteam-bs [20:54] *** brayden has joined #archiveteam-bs [20:56] *** HCross has quit IRC (Read error: Connection reset by peer) [20:56] *** xit has quit IRC (Read error: Connection reset by peer) [20:56] *** horkermon has quit IRC (Read error: Connection reset by peer) [20:57] *** justcool3 has joined #archiveteam-bs [20:58] *** hook54321 has joined #archiveteam-bs [21:03] *** HCross has joined #archiveteam-bs [21:07] *** horkermon has joined #archiveteam-bs [21:09] *** ThisAsYou has quit IRC (Read error: Connection reset by peer) [21:10] *** prq has quit IRC (Read error: Operation timed out) [21:10] *** mtntmnky_ has quit IRC (Read error: Operation timed out) [21:10] *** nyany has quit IRC (Write error: Broken pipe) [21:10] *** revi has quit IRC (Ping timeout: 260 seconds) [21:10] *** jrwr has quit IRC (Ping timeout: 260 seconds) [21:11] *** Pixi` has joined #archiveteam-bs [21:12] *** phirephl- has quit IRC (Read error: Operation timed out) [21:13] *** svchfoo1 has quit IRC (Read error: Operation timed out) [21:13] *** Igloo has quit IRC (Read error: Operation timed out) [21:13] *** nico_32_ has joined #archiveteam-bs [21:13] *** nico_32 has quit IRC (Read error: Operation timed out) [21:14] *** hook54321 has quit IRC () [21:14] *** Pixi has quit IRC (Read error: Operation timed out) [21:14] *** Iglooop1 has quit IRC (Read error: Operation timed out) [21:15] *** hook54321 has joined #archiveteam-bs [21:16] *** horkermon has quit IRC (Read error: Connection reset by peer) [21:17] *** phirephly has joined #archiveteam-bs [21:17] *** DigiDigi has quit IRC (Read error: Operation timed out) [21:17] *** pew has quit IRC (Ping timeout: 265 seconds) [21:22] *** fallenoak has joined #archiveteam-bs [21:25] *** qw3rty__ has joined #archiveteam-bs [21:26] *** jodizzle_ is now known as jodizzle [21:27] *** xit has joined #archiveteam-bs [21:27] *** asdf0101 has quit IRC (Read error: Operation timed out) [21:27] *** asdf0101 has joined #archiveteam-bs [21:28] *** DigiDigi has joined #archiveteam-bs [21:29] *** nyany has joined #archiveteam-bs [21:29] *** prq has joined #archiveteam-bs [21:29] *** svchfoo1 has joined #archiveteam-bs [21:29] *** svchfoo3 sets mode: +o svchfoo1 [21:30] *** Iglooop1 has joined #archiveteam-bs [21:30] *** revi has joined #archiveteam-bs [21:31] *** xit has quit IRC (Client Quit) [21:32] *** qw3rty_ has quit IRC (Read error: Operation timed out) [21:34] *** jrwr has joined #archiveteam-bs [21:34] *** Igloo has joined #archiveteam-bs [21:36] *** horkermon has joined #archiveteam-bs [21:37] *** hook54321 has quit IRC () [21:37] *** mtntmnky_ has joined #archiveteam-bs [21:38] *** hook54321 has joined #archiveteam-bs [21:39] I give up; I'll zip up the metadata I gathered, upload it, and stop spending time on this :p [21:39] *** ThisAsYou has joined #archiveteam-bs [21:40] *** ThisAsYou has quit IRC (Read error: Connection reset by peer) [21:41] *** Darkstar has quit IRC (http://quassel-irc.org - Chat comfortably. Anywhere.) [21:46] *** picklefac has joined #archiveteam-bs [21:46] *** BnAboyZ has joined #archiveteam-bs [21:49] *** hook54321 has quit IRC (Read error: Connection reset by peer) [21:52] *** ThisAsYou has joined #archiveteam-bs [21:55] *** chfoo has quit IRC (Read error: Operation timed out) [21:55] http://data.nicolas17.xyz/reckful/reckful-clips-meta.ndjson.gz [21:55] *** chfoo has joined #archiveteam-bs [21:56] *** fallenoak has quit IRC (Read error: Connection reset by peer) [21:57] *** HCross has quit IRC (Read error: Connection reset by peer) [22:01] *** mattl has joined #archiveteam-bs [22:03] *** fallenoak has joined #archiveteam-bs [22:04] *** benjinsmi has joined #archiveteam-bs [22:07] *** benjinss has quit IRC (Ping timeout: 265 seconds) [22:08] *** HCross has joined #archiveteam-bs [22:08] *** Darkstar has joined #archiveteam-bs [22:09] *** mattl has quit IRC (Read error: Connection reset by peer) [22:09] *** justcool3 has quit IRC (Read error: Connection reset by peer) [22:16] *** horkermon has quit IRC (Read error: Connection reset by peer) [22:22] *** horkermon has joined #archiveteam-bs [22:22] *** britmob_ has joined #archiveteam-bs [22:22] *** BlueMax has quit IRC (Read error: Connection reset by peer) [22:23] *** horkermon has quit IRC (Read error: Connection reset by peer) [22:23] *** britmob_ has quit IRC (Remote host closed the connection) [22:25] *** britmob_ has joined #archiveteam-bs [22:25] *** horkermon has joined #archiveteam-bs [22:25] *** OrIdow6 has quit IRC (Remote host closed the connection) [22:25] *** benjinsmi has quit IRC (Ping timeout: 265 seconds) [22:25] *** britmob has quit IRC (Ping timeout: 265 seconds) [22:26] *** yano has joined #archiveteam-bs [22:26] *** benjins has joined #archiveteam-bs [22:26] *** yano_ has quit IRC (Read error: Connection reset by peer) [22:26] *** Zerote has quit IRC (Read error: Connection reset by peer) [22:26] *** SJon___ has quit IRC (Ping timeout: 265 seconds) [22:27] *** Zerote has joined #archiveteam-bs [22:27] *** OrIdow6 has joined #archiveteam-bs [22:28] *** SJon___ has joined #archiveteam-bs [22:28] *** mattl has joined #archiveteam-bs [22:29] *** justcool3 has joined #archiveteam-bs [22:36] *** hook54321 has joined #archiveteam-bs [22:44] *** qwebirc94 has joined #archiveteam-bs [22:46] *** G4llifrey has joined #archiveteam-bs [22:46] *** qwebirc94 has left [22:47] *** qwebirc56 has joined #archiveteam-bs [22:47] *** qwebirc56 has quit IRC (Client Quit) [22:52] *** Gallifrey has left [22:52] *** Ctrl-S___ has joined #archiveteam-bs [22:54] *** Vito` has joined #archiveteam-bs [22:55] *** amelia386 has joined #archiveteam-bs [22:58] *** diggan has joined #archiveteam-bs [22:58] *** pnJay has joined #archiveteam-bs [22:58] *** DrasticAc has joined #archiveteam-bs [22:58] *** jesse-s has joined #archiveteam-bs [22:58] *** abartov__ has joined #archiveteam-bs [22:58] *** riking_ has joined #archiveteam-bs [22:58] *** tchaypo_ has joined #archiveteam-bs [22:59] *** c0mpass has joined #archiveteam-bs [23:08] *** Arcorann_ has joined #archiveteam-bs [23:28] *** pew has joined #archiveteam-bs [23:44] *** WalkFly has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [23:46] *** hook54321 has quit IRC () [23:47] *** hook54321 has joined #archiveteam-bs