[00:07] *** netsound has quit IRC (Remote host closed the connection) [00:11] *** enowaldo has joined #archiveteam-bs [00:16] *** enowaldo has quit IRC (Read error: Operation timed out) [00:17] *** Frogging has joined #archiveteam-bs [00:20] *** JSharp has quit IRC (Read error: Connection reset by peer) [00:28] *** revi has quit IRC (Read error: Connection reset by peer) [00:33] *** killsushi has quit IRC (Quit: Leaving) [00:42] *** revi has joined #archiveteam-bs [00:42] *** JSharp has joined #archiveteam-bs [00:42] *** omarroth has quit IRC (Ping timeout: 506 seconds) [00:43] *** omarroth has joined #archiveteam-bs [00:49] *** omarroth has quit IRC (Quit: Konversation terminated!) [01:14] *** killsushi has joined #archiveteam-bs [01:18] *** BlueMax has quit IRC (Quit: Leaving) [01:59] *** Despatche has quit IRC (Quit: Read error: Connection reset by deer) [02:12] *** BlueMax has joined #archiveteam-bs [02:33] *** enowaldo has joined #archiveteam-bs [02:38] *** Zerote has quit IRC (Ping timeout: 260 seconds) [02:39] *** enowaldo has quit IRC (Ping timeout: 265 seconds) [03:08] *** Mateon1 has quit IRC (Remote host closed the connection) [03:08] *** Mateon1 has joined #archiveteam-bs [03:11] *** wyatt8740 has quit IRC (Read error: Operation timed out) [03:20] *** wyatt8740 has joined #archiveteam-bs [03:28] *** qw3rty117 has joined #archiveteam-bs [03:30] *** killsushi has quit IRC (Quit: Leaving) [03:30] *** wyatt8740 has quit IRC (Read error: Connection reset by peer) [03:31] *** wyatt8740 has joined #archiveteam-bs [03:34] *** odemgi_ has joined #archiveteam-bs [03:35] *** qw3rty116 has quit IRC (Read error: Operation timed out) [03:36] *** odemgi has quit IRC (Read error: Operation timed out) [03:43] *** odemg has quit IRC (Ping timeout: 615 seconds) [03:50] *** odemg has joined #archiveteam-bs [04:24] *** tech234a has joined #archiveteam-bs [04:35] *** enowaldo has joined #archiveteam-bs [04:44] *** enowaldo has quit IRC (Read error: Operation timed out) [05:14] *** enowaldo has joined #archiveteam-bs [05:19] *** enowaldo has quit IRC (Ping timeout: 265 seconds) [05:29] *** Exairnous has quit IRC (Ping timeout: 615 seconds) [05:50] *** ndiddy has quit IRC () [05:55] *** Odd0002_ has joined #archiveteam-bs [06:00] *** Odd0002 has quit IRC (Ping timeout: 615 seconds) [06:00] *** Odd0002_ is now known as Odd0002 [06:23] *** Dimtree has quit IRC () [06:34] *** tech234a has quit IRC (Quit: Connection closed for inactivity) [06:36] *** Dimtree has joined #archiveteam-bs [07:08] *** wabu has quit IRC (Read error: Operation timed out) [07:10] *** wabu has joined #archiveteam-bs [07:29] *** purplebot has quit IRC (Read error: Operation timed out) [07:29] *** sknebel has quit IRC (Read error: Connection reset by peer) [07:29] *** purplebot has joined #archiveteam-bs [07:30] *** Ing3b0rg has quit IRC (Ping timeout: 506 seconds) [07:30] *** sknebel has joined #archiveteam-bs [07:30] *** benjins has joined #archiveteam-bs [07:32] *** Ing3b0rg has joined #archiveteam-bs [07:32] *** colona_ has joined #archiveteam-bs [07:32] *** benjinss has quit IRC (Ping timeout: 506 seconds) [07:33] *** wp494 has quit IRC (Ping timeout: 506 seconds) [07:35] *** wp494 has joined #archiveteam-bs [07:35] *** Fusl has quit IRC (Ping timeout: 740 seconds) [07:35] *** fratti has joined #archiveteam-bs [07:36] *** Coderjo_ has joined #archiveteam-bs [07:36] *** Coderjo has quit IRC (Read error: Connection reset by peer) [07:36] *** Fusl has joined #archiveteam-bs [07:36] *** fratti_ has quit IRC (Ping timeout: 506 seconds) [07:37] *** svchfoo1 sets mode: +o Fusl [07:37] *** svchfoo3 sets mode: +o Fusl [07:38] *** purplebot has quit IRC (Ping timeout: 506 seconds) [07:38] *** purplebot has joined #archiveteam-bs [07:38] *** colona has quit IRC (Ping timeout: 740 seconds) [07:56] *** purplebot has quit IRC (Read error: Connection reset by peer) [07:56] *** purplebot has joined #archiveteam-bs [08:01] *** closure has quit IRC (Read error: Connection reset by peer) [08:01] *** closure has joined #archiveteam-bs [08:04] *** C4K3 has quit IRC (Ping timeout: 506 seconds) [08:08] *** C4K3 has joined #archiveteam-bs [08:12] *** eythian has quit IRC (Ping timeout: 740 seconds) [08:19] *** eythian has joined #archiveteam-bs [08:23] *** ugh has joined #archiveteam-bs [08:23] *** legoktm has quit IRC (Read error: Connection reset by peer) [08:23] *** sknebel has quit IRC (Read error: Connection reset by peer) [08:25] *** sknebel has joined #archiveteam-bs [08:43] *** Despatche has joined #archiveteam-bs [09:34] I reran my Sola API URL extraction with a better regex (yay for "parsing" JSON with regex!) and got a few 100k additional URLs. 8.73 million now. I'll start grabbing those shortly. I'll grab the Sola CDN stuff on my machine and throw everything external into ArchiveBot. [09:34] This list includes all URLs that appear within posts, i.e. outlinks from Sola. [09:54] *** noirscape has joined #archiveteam-bs [09:55] *** JAA has quit IRC (Read error: Operation timed out) [09:56] *** Frogging has quit IRC (Read error: Operation timed out) [09:56] *** cfarquhar has quit IRC (Read error: Operation timed out) [09:56] *** Frogging has joined #archiveteam-bs [09:56] *** wabu has quit IRC (Read error: Operation timed out) [09:57] *** svchfoo1 has quit IRC (Read error: Operation timed out) [09:58] *** c4rc4s has quit IRC (Read error: Operation timed out) [09:58] *** simon816 has quit IRC (Ping timeout: 246 seconds) [09:58] *** eythian has quit IRC (Read error: Operation timed out) [10:00] *** wp494 has quit IRC (Ping timeout: 493 seconds) [10:01] *** Mateon1 has quit IRC (Ping timeout: 493 seconds) [10:01] *** Mateon1 has joined #archiveteam-bs [10:02] *** wp494 has joined #archiveteam-bs [10:18] *** eythian has joined #archiveteam-bs [10:34] *** wacky has quit IRC (Ping timeout: 615 seconds) [10:45] *** Verified_ has quit IRC (Ping timeout: 252 seconds) [10:52] *** Hani111 has joined #archiveteam-bs [10:57] *** svchfoo1 has joined #archiveteam-bs [10:57] *** Fusl sets mode: +o svchfoo1 [10:57] *** simon816 has joined #archiveteam-bs [10:57] *** c4rc4s has joined #archiveteam-bs [10:57] *** wacky has joined #archiveteam-bs [10:58] *** cfarquhar has joined #archiveteam-bs [10:59] *** Verified_ has joined #archiveteam-bs [11:00] *** JAA has joined #archiveteam-bs [11:00] *** Fusl sets mode: +o JAA [11:00] *** bakJAA sets mode: +o JAA [11:01] *** Hani has quit IRC (Ping timeout: 615 seconds) [11:01] *** Hani111 is now known as Hani [11:01] *** wabu has joined #archiveteam-bs [11:15] *** enowaldo has joined #archiveteam-bs [11:16] *** Stiletto has quit IRC (Ping timeout: 265 seconds) [11:28] *** enowaldo has quit IRC (Read error: Operation timed out) [11:43] *** Sian1468 has joined #archiveteam-bs [11:45] JAA: i've been archiving http://srv.deutschlandradio.de/aodlistaudio.1706.de.rpc for about two years now, just shout if you want the files (via rclone, i have them in gc) [11:46] *** Sian1468 has quit IRC (Client Quit) [11:51] schbirid: Come to #radio-archive and talk to arkiver! [12:12] hm, that seems more like live radiu [12:12] what i archive is more "mediatheken" (yay germany) [12:12] on-demand stuff [12:16] Ah, sorry, just read Deutschland Radio and "archiving for two years" and assumed it was the live stream without checking. [12:16] Any idea about the size? [12:18] true, that was misleading [12:18] let me have a look [12:23] *** icedice has joined #archiveteam-bs [12:28] this will take a while =) [12:37] *** BlueMax has quit IRC (Quit: Leaving) [12:44] I've started my Sola CDN retrieval. 6.52 million URLS on cdn.solacore.net and s3.amazonaws.com. [12:46] load average: 8.46, 8.51, 5.63 [12:47] Time for this machine to do some work. [12:52] *** omarroth has joined #archiveteam-bs [12:58] *** bitBaron has joined #archiveteam-bs [13:09] *** Hani111 has joined #archiveteam-bs [13:10] *** Hani has quit IRC (Read error: Operation timed out) [13:14] *** Hani111 has quit IRC (Ping timeout: 255 seconds) [13:14] *** Hani has joined #archiveteam-bs [13:29] *** omarroth has quit IRC (Remote host closed the connection) [13:31] *** omarroth has joined #archiveteam-bs [13:32] *** enowaldo has joined #archiveteam-bs [13:49] *** Zerote has joined #archiveteam-bs [13:53] *** overflowe has quit IRC (Remote host closed the connection) [13:54] *** enowaldo has quit IRC (Ping timeout: 265 seconds) [14:07] *** omarroth has quit IRC (Ping timeout: 246 seconds) [14:08] *** omarroth has joined #archiveteam-bs [14:10] *** bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…) [14:17] *** bitBaron has joined #archiveteam-bs [14:32] *** Sian1468 has joined #archiveteam-bs [14:37] *** Sian1468 has quit IRC (Client Quit) [14:38] *** argus has joined #archiveteam-bs [15:18] *** Hani111 has joined #archiveteam-bs [15:19] *** Hani111_ has joined #archiveteam-bs [15:21] *** Hani111__ has joined #archiveteam-bs [15:23] *** bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…) [15:23] *** omarroth has quit IRC (Read error: Connection reset by peer) [15:25] Sola CDN: 3 hours, about 10 % done, so I expect it to finish tomorrow afternoon. [15:27] *** Hani has quit IRC (Ping timeout: 615 seconds) [15:27] *** Hani111__ is now known as Hani [15:29] *** Hani111 has quit IRC (Ping timeout: 615 seconds) [15:31] *** Hani111_ has quit IRC (Ping timeout: 615 seconds) [15:39] *** Rome_Silv has joined #archiveteam-bs [15:47] *** VADemon has joined #archiveteam-bs [15:50] ivan, I've just received a notice that a YT channel has been geoblocked in DE (politics) can you please archive it? link: [15:50] https://www.youtube.com/channel/UC4teOyQNXzMH94YMYySl9mA [15:53] grabbing [15:55] *** icedice has quit IRC (Quit: Leaving) [16:37] *** VerifiedJ has joined #archiveteam-bs [17:02] *** VerifiedJ has quit IRC (Read error: Connection reset by peer) [17:02] *** VerifiedJ has joined #archiveteam-bs [17:06] *** VADemon has quit IRC (Quit: Bye) [17:06] *** VADemon has joined #archiveteam-bs [17:08] *** VerifiedJ has quit IRC (Ping timeout: 252 seconds) [17:08] How did I not know about this? http://webarchive.loc.gov/ [17:09] *** VerifiedJ has joined #archiveteam-bs [17:12] I guess you knew https://www.webarchive.org.uk/ [17:12] *** VerifiedJ has quit IRC (Read error: Connection reset by peer) [17:14] Yeah [17:14] *** enowaldo has joined #archiveteam-bs [17:15] *** godane has quit IRC (Quit: Leaving.) [17:19] *** enowaldo has quit IRC (Ping timeout: 268 seconds) [17:20] *** bitBaron has joined #archiveteam-bs [17:25] *** godane has joined #archiveteam-bs [17:36] *** bitBaron has quit IRC (My computer has gone to sleep. 😴😪ZZZzzz…) [17:37] *** bitBaron has joined #archiveteam-bs [18:02] JAA: ~2TB [18:02] ~50G per month [18:06] *** enowaldo has joined #archiveteam-bs [18:07] *** ndiddy has joined #archiveteam-bs [18:11] *** Verified_ has quit IRC (Ping timeout: 252 seconds) [18:16] *** Verified_ has joined #archiveteam-bs [18:24] *** Verified_ has quit IRC (Ping timeout: 252 seconds) [18:36] *** Verified_ has joined #archiveteam-bs [18:45] *** killsushi has joined #archiveteam-bs [18:50] schbirid: WARCs? [18:59] *** bitBaron has quit IRC (My computer has gone to sleep. 😴😪ZZZzzz…) [19:38] wpull seg faulting, always a pleasure. [19:44] *** RomeSilva has joined #archiveteam-bs [19:46] *** Rome_Silv has quit IRC (Ping timeout: 252 seconds) [19:51] *** cfarquhar has quit IRC (Read error: Operation timed out) [19:56] *** cfarquhar has joined #archiveteam-bs [19:57] *** enowaldo has quit IRC (Read error: Operation timed out) [19:58] JAA: hell no, just the media files :) [19:59] Aw ;-) [19:59] Did you keep metadata as well? [20:12] nope :D [20:12] :-( [20:14] Is there a way to get the relevant metadata now? [20:16] dont think so [20:16] now i feel bad :D [20:16] i should simply zip up the feed pages daily [20:16] the files are nicely named and are in a y/m/d structure on the originating server though so it is pretty good [20:26] Looks like it should be possible to collect the information. [20:26] E.g. https://srv.deutschlandradio.de/aodlistaudio.1706.de.rpc?drau:station_id=4&drau:from=01.10.2018&drau:to=01.10.2018&drau:page=1&drau:limit=30 [20:29] Or rather something like https://srv.deutschlandradio.de/aodlistaudio.1706.de.rpc?drau:page=1&drau:limit=30 [20:30] 995 pages with limit=100. [20:32] *** enowaldo has joined #archiveteam-bs [20:41] *** enowaldo has quit IRC (Ping timeout: 492 seconds) [21:05] *** bitBaron has joined #archiveteam-bs [21:25] *** Rome has joined #archiveteam-bs [21:31] *** RomeSilva has quit IRC (Read error: Operation timed out) [21:41] *** wabu has quit IRC (Read error: Operation timed out) [21:42] *** cfarquhar has quit IRC (Read error: Operation timed out) [21:42] *** Exairnous has joined #archiveteam-bs [21:42] *** svchfoo1 has quit IRC (Read error: Operation timed out) [21:44] *** wm_ has joined #archiveteam-bs [21:44] *** simon816 has quit IRC (Ping timeout: 246 seconds) [21:44] *** c4rc4s has quit IRC (Read error: Operation timed out) [21:45] *** JAA has quit IRC (Ping timeout: 246 seconds) [21:45] *** Rome has quit IRC (Remote host closed the connection) [21:45] *** Rome has joined #archiveteam-bs [22:15] *** tech234a has joined #archiveteam-bs [22:43] *** simon816 has joined #archiveteam-bs [22:43] *** c4rc4s has joined #archiveteam-bs [22:43] *** svchfoo1 has joined #archiveteam-bs [22:43] *** Fusl sets mode: +o svchfoo1 [22:43] *** cfarquhar has joined #archiveteam-bs [22:44] *** JAA has joined #archiveteam-bs [22:44] *** Fusl sets mode: +o JAA [22:44] *** bakJAA sets mode: +o JAA [22:46] *** wabu has joined #archiveteam-bs [23:26] *** ndiddy has quit IRC (Ping timeout: 252 seconds) [23:28] *** BlueMax has joined #archiveteam-bs [23:34] *** Damme_ has joined #archiveteam-bs [23:36] *** Atom-- has joined #archiveteam-bs [23:36] *** benjinsmi has joined #archiveteam-bs [23:36] *** dxrt- has joined #archiveteam-bs [23:36] *** ats_ has joined #archiveteam-bs [23:39] *** sep332_ has joined #archiveteam-bs [23:41] *** bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…) [23:41] *** zino has quit IRC (Quit: Leaving) [23:41] *** zino has joined #archiveteam-bs [23:42] *** svchfoo1 sets mode: +o zino [23:42] *** svchfoo3 sets mode: +o zino [23:42] *** joepie91_ has joined #archiveteam-bs [23:43] *** Rome has quit IRC (Read error: Connection reset by peer) [23:44] *** Rome has joined #archiveteam-bs [23:44] *** Verified_ has quit IRC (se.hub irc.underworld.no) [23:44] *** C4K3 has quit IRC (se.hub irc.underworld.no) [23:44] *** benjins has quit IRC (se.hub irc.underworld.no) [23:44] *** Damme has quit IRC (se.hub irc.underworld.no) [23:44] *** joepie91 has quit IRC (se.hub irc.underworld.no) [23:44] *** ats has quit IRC (se.hub irc.underworld.no) [23:44] *** sep332 has quit IRC (se.hub irc.underworld.no) [23:44] *** Atom__ has quit IRC (se.hub irc.underworld.no) [23:44] *** dxrt has quit IRC (se.hub irc.underworld.no) [23:44] *** jut has quit IRC (se.hub irc.underworld.no) [23:44] *** coderobe has quit IRC (se.hub irc.underworld.no) [23:44] *** ColdIce has quit IRC (se.hub irc.underworld.no) [23:44] *** kiska has quit IRC (se.hub irc.underworld.no) [23:44] *** ranma has quit IRC (se.hub irc.underworld.no) [23:44] *** Flashfire has quit IRC (se.hub irc.underworld.no) [23:44] *** i0npulse has quit IRC (se.hub irc.underworld.no) [23:48] *** C4K3_ has joined #archiveteam-bs [23:50] Sola CDN: 11 hours, 38 % done, still on track for ~30 hour total [23:55] *** enowaldo has joined #archiveteam-bs [23:56] Or not since my disks are filling up. [23:56] 400 GiB already...