[00:05] *** PotcFdk has quit IRC (~'o'/) [00:05] Atluxity: I didn't have time yet [00:05] will fix it now [00:13] Please force stop anything you have running for oldfriends! [00:14] Scripts are updated. Please update your oldfriends scripts. [00:24] *** PotcFdk has joined #archiveteam [00:30] arkiver: I give up... │··················· [00:30] Process WgetDownload returned exit code -6 for Item 10institutions:7886 │··················· [00:30] Retrying WgetDownload for Item 10institutions:7886 after 30 seconds... [00:30] I give up was from the commandline, not me lol [00:43] *** MrRadar has quit IRC (Read error: Operation timed out) [00:50] hmm [00:50] the site seems to work [00:51] They blocked our user agent [00:52] *** zer0rest has joined #archiveteam [00:52] *** MrRadar has joined #archiveteam [00:54] Oldfriends scripts updated! Now using a firefox useragent [00:56] Be warned: they might ban IPs [00:56] How rude. [01:21] *** slyphic|a is now known as slyphic [01:27] *** zer0rest has quit IRC (Read error: Connection reset by peer) [01:38] *** Ghost_of_ has quit IRC (Quit: Leaving) [01:41] *** JesseW has joined #archiveteam [01:54] *** ats has quit IRC (Ping timeout: 252 seconds) [02:00] *** philpem has quit IRC (Ping timeout: 260 seconds) [02:06] *** ats has joined #archiveteam [02:08] *** Ravenloft has quit IRC (Read error: Connection reset by peer) [02:11] *** username1 has joined #archiveteam [02:13] *** schbirid2 has quit IRC (Read error: Operation timed out) [02:31] *** REiN^ has quit IRC (Read error: Operation timed out) [02:45] *** JesseW has quit IRC (Ping timeout: 615 seconds) [02:51] *** JesseW has joined #archiveteam [02:54] arkiver: I think they did, still getting "Server returned 0 (HERR). Sleeping." for everything [02:56] *** W1nterFox has joined #archiveteam [03:00] *** WinterFox has quit IRC (Read error: Operation timed out) [03:02] *** Ravenloft has joined #archiveteam [03:03] *** BlueMaxim has quit IRC (Quit: Leaving) [03:06] *** JesseW has quit IRC (Leaving.) [03:46] *** vitzli has joined #archiveteam [03:50] *** BlueMaxim has joined #archiveteam [04:41] *** turnkit|2 has joined #archiveteam [04:41] *** JesseW has joined #archiveteam [04:42] *** no2penci1 has joined #archiveteam [04:42] *** no2pencil has quit IRC (Read error: Operation timed out) [04:43] *** bai_ has joined #archiveteam [04:45] *** vOYtEC has quit IRC (hub.efnet.us irc.Prison.NET) [04:45] *** turnkit has quit IRC (hub.efnet.us irc.Prison.NET) [04:45] *** lbft has quit IRC (hub.efnet.us irc.Prison.NET) [04:45] *** bai has quit IRC (hub.efnet.us irc.Prison.NET) [04:45] *** dtm has quit IRC (hub.efnet.us irc.Prison.NET) [04:50] *** lbft_ has joined #archiveteam [05:16] *** bai_ is now known as bai [05:33] *** megaminxw has quit IRC (Quit: Leaving.) [05:55] Ha ha, Oldfriends just yelled at me [05:55] guess you're not friends anymore [05:56] Congrats, you evil little shits [05:56] "Oh, you want us to stop hogging your precious dying server's bandwidth? You can mail your database to 300 Funston Ave, San Francisco CA 94118." [06:04] *** bentpins has joined #archiveteam [06:07] Hi there [06:07] We noticed a spike in requests coming through to OldFriends.co.nz site yesterday morning (NZ time), with the requests coming from .archive team.. [06:07] I see OldFriends is currently on the archive teams warriors projects. [06:07] At the moment we.ve blocked the requests as they have been putting pressure on our CPUs - in turn affecting some other systems which run on the same kit as OldFriends. [06:07] The bulk of URIs being requested look to be malformed / garbage requests. [06:07] [06:07] We are shutting OldFriends.co.nz, but just letting you know we are actively working with the Alexander Turnbill library (NZ National Library) to get the public content archived for public consumption once the site closes. [06:07] Let me know if you.ve got any questions [06:07] [06:07] Sean Cresswell [06:07] [06:08] Test Manager | Trade Me [06:08] 021 554 083 [06:08] www.trademe.co.nz [06:10] * JesseW just switched my warrior over to oldfriends [06:11] make sure it's latest version of the script [06:11] Is there any way to reduce the "malformed" requests? [06:11] restarted the warrior just before [06:11] that works [06:11] https://github.com/ArchiveTeam/oldfriends-grab/commit/9a6b31b8c0f5861428d6a8432d7d71300335dd5a fixes the %2c%2c... bit [06:11] I think [06:12] and there are apparently few nicknames doing this -- I immediately made it on to the top ten list [06:30] *** JesseW has quit IRC (Leaving.) [06:33] *** dtm has joined #archiveteam [06:52] *** Atom__ has quit IRC (Read error: Connection reset by peer) [06:52] *** logan2 has joined #archiveteam [06:53] *** logan has quit IRC (Ping timeout: 252 seconds) [07:07] Small world, I know people in trademe. If it does make it to the library offline, I'll definately have a go at getting a copy [07:12] I'm firing up oldfriends-grab again [07:18] it seem to me like our targets give us more slack the better our crawling is. [07:27] *** Ravenloft has quit IRC (Read error: Connection reset by peer) [07:36] *** X1011 has joined #archiveteam [07:58] and then some are just asses [08:05] *** REiN^ has joined #archiveteam [08:41] *** vitzli has quit IRC (Quit: Leaving) [09:43] *** Ghost_of_ has joined #archiveteam [09:49] *** bentpins has quit IRC (Read error: Operation timed out) [09:51] *** atomotic has joined #archiveteam [09:54] *** zer0rest has joined #archiveteam [10:04] Yeah, the problem with %2c%2c should be fixed [10:06] The grab is restarted anyway and is almost finished [10:06] *** Stiletto has quit IRC (Read error: Operation timed out) [10:10] *** nertzy has joined #archiveteam [10:11] SketchCow: maybe you can reply to oldfriends that we identified the problem as a loop with %2c%2c and have fixed it. The grab is almost finished [10:12] *** vitzli has joined #archiveteam [10:26] *** Stiletto has joined #archiveteam [10:33] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [10:50] arkiver: these are the last items? [10:50] Problably [10:51] nice [10:51] Thing is, there's also some items that we csan't easily make alist of without scnning the site [10:51] so I'll also add it to archivebot with some rules [10:53] *** nertzy has quit IRC (Quit: This computer has gone to sleep) [11:20] *** Ghost_of_ has quit IRC (Quit: Leaving) [12:37] *** megaminxw has joined #archiveteam [12:54] *** BlueMaxim has quit IRC (Quit: Leaving) [12:59] http://tvdags.se/artikel/tvdags-avslojar-stora-forandringar-for-utbudet-i-svts-oppet-arkiv (Swedish Television is going to start pruning oppetarkiv.se ("Open Archive.se") because of new way of licensing the content) [13:00] Pruning starts at Jan 13th [13:30] *** W1nterFox has quit IRC (Remote host closed the connection) [13:34] *** Atom__ has joined #archiveteam [13:40] Might be be problematic to archive though, since the content is geolocked to swedish IPs [13:41] Anyone know of a cloud provider with a swedish datacenter? [13:42] phuzion: There's https://glesys.se/vps for one [13:43] Or maybe you wanted one located in england/us but with DCs in Sweden? [13:43] Doesn't really matter, as long as the price isn't horrible. [13:44] The idea was to get a handful of machines up with swedish IPs if necessary. [13:44] Or hell, we could find a swedish VPN provider and use that. [13:46] Then there's https://mullvad.net/en/ https://www.ovpn.se/ https://anonine.com/en/ [13:51] There's multiple Swedish TOR exits, not sure if it's possible to pin exit by country [13:51] there's a few swedish users here (me, tobbez and a few others) [13:52] there's a svtplay-dl tool (pretty much like youtube-dl) which can dl from oppetarkiv.se (not WARCed tho) [14:22] *** SadDM has joined #archiveteam [14:22] *** swebb sets mode: +o SadDM [14:46] youtube-dl works too [14:47] *** Start has quit IRC (Quit: Disconnected.) [14:50] *** Atom-- has joined #archiveteam [14:53] *** Atom__ has quit IRC (Ping timeout: 252 seconds) [15:32] *** atomotic has joined #archiveteam [15:37] If I contact them it will agitate them. [15:37] We'll grab it all and then deal. [15:39] *** zer0rest has quit IRC (Ping timeout: 260 seconds) [15:41] *** zer0rest has joined #archiveteam [15:41] SketchCow: ok (I guess your talking about oldfriends) [15:43] I like your negotiation style [15:43] *** vitzli has quit IRC (Quit: Leaving) [15:44] http://gmc.yoyogames.com/index.php?showtopic=687124 [15:46] *** zer0rest has quit IRC (Ping timeout: 260 seconds) [16:02] *** Start has joined #archiveteam [16:06] *** zer0rest has joined #archiveteam [16:07] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [16:07] *** arkiver2 has joined #archiveteam [16:14] *** zer0rest has quit IRC (Read error: Connection reset by peer) [16:25] *** zer0rest has joined #archiveteam [16:30] *** zer0rest has quit IRC (Read error: Connection reset by peer) [16:30] *** zer0rest has joined #archiveteam [16:34] *** zer0rest has quit IRC (Read error: Connection reset by peer) [16:34] *** zer0rest has joined #archiveteam [16:36] *** zer0rest has quit IRC (Read error: Connection reset by peer) [16:37] *** zer0rest has joined #archiveteam [16:42] *** megaminxw has quit IRC (Quit: Leaving.) [16:44] *** megaminxw has joined #archiveteam [16:45] *** megaminxw has quit IRC (Client Quit) [16:45] *** zer0rest has quit IRC (Read error: Connection reset by peer) [16:46] *** zer0rest has joined #archiveteam [16:47] *** zer0rest has quit IRC (Read error: Connection reset by peer) [16:48] *** zer0rest has joined #archiveteam [16:51] *** zer0rest has quit IRC (Read error: Connection reset by peer) [17:07] *** Start has quit IRC (Quit: Disconnected.) [17:08] *** zer0rest has joined #archiveteam [17:13] *** Start has joined #archiveteam [17:13] *** zer0rest has quit IRC (Read error: Connection reset by peer) [17:13] *** zer0rest has joined #archiveteam [17:18] *** zer0rest has quit IRC (Read error: Connection reset by peer) [17:21] *** zer0rest has joined #archiveteam [17:23] *** Start has quit IRC (Quit: Disconnected.) [17:24] *** BubuAnabe has joined #archiveteam [17:27] *** BubuAnabe has quit IRC (http://www.kiwiirc.com/ - A hand crafted IRC client) [17:28] *** BubuAnabe has joined #archiveteam [17:29] Hi, I'd like to join/help you, and there's a website that may be deleted soon by the new goverment and I'd like to archive it and doesn't know really how. [17:32] which website? [17:32] http://nacionalrock.com/ [17:32] *** nertzy has joined #archiveteam [17:34] I'd do an archivebot, then grab whatever channel has all those youtubes [17:34] It's a wordpress site. It's a public radio station with great journalist and musicians and it has been closed by the new government. [17:34] https://www.youtube.com/channel/UCW3CLjm-lNO9_RtZudI4Rgg [17:35] I'm alredy downloading youtube stuff [17:38] Good. [17:38] *** RichardG has quit IRC (Ping timeout: 255 seconds) [17:39] The most important stuff of the website are all the media files, like audios uploaded all in http://www.nacionalrock.com/wp-content/uploads/ [17:39] But have to parse it all scaning all the blog posts [17:40] btw u'r great haha thank you so much [17:43] *** RichardG has joined #archiveteam [17:43] Are you downloading youtube using a good method or just winging it [17:43] We're pretty good at that [17:44] i'm making a web archive of ?p= urls [17:44] JDownloader all at the best quality. I think 1080p vid comes without audio, so I'll be mergin then with ffmpeg later. [17:44] only so we can grab the mp3s better [17:46] jdownloader works for the video but I don't think it gets the metadata [17:46] we have some notes at http://www.archiveteam.org/index.php?title=Youtube [17:47] youtube-dl with the appropriate parameters will get all the metadata and call out to ffmpeg to merge the video and audio [17:47] Ok, I'll check it right away! [17:54] *** BubuAnabe has quit IRC (http://www.kiwiirc.com/ - A hand crafted IRC client) [17:56] *** BubuAnabe has joined #archiveteam [17:56] My internet connection is being hell right now, I'm not sure if I may be able to download the youtube videos, damn! [17:57] *** philpem has joined #archiveteam [17:57] *** HCross has quit IRC (Read error: Connection reset by peer) [18:05] *** HCross has joined #archiveteam [18:06] *** arkiver2 has quit IRC (Ping timeout: 260 seconds) [18:12] *** nertzy has quit IRC (Quit: This computer has gone to sleep) [18:12] *** zer0rest has quit IRC (Read error: Connection reset by peer) [18:20] *** zer0rest has joined #archiveteam [18:32] *** zer0rest has quit IRC (Read error: Connection reset by peer) [18:32] *** zer0rest has joined #archiveteam [18:40] *** zer0rest has quit IRC (Ping timeout: 260 seconds) [18:41] *** Start has joined #archiveteam [18:42] *** brayden has quit IRC (Ping timeout: 606 seconds) [18:49] *** cadbury_ has joined #archiveteam [18:53] *** cadbury has quit IRC (Read error: Connection reset by peer) [19:07] *** BubuAnabe has quit IRC (http://www.kiwiirc.com/ - A hand crafted IRC client) [19:16] *** nertzy has joined #archiveteam [19:19] *** Start has quit IRC (Quit: Disconnected.) [19:23] *** Start has joined #archiveteam [19:24] *** BubuAnabe has joined #archiveteam [19:27] *** bwn has joined #archiveteam [19:28] *** K4k has joined #archiveteam [19:28] *** K4k has quit IRC (Connection closed) [19:28] *** K4k has joined #archiveteam [19:28] *** K4k has quit IRC (Connection closed) [19:29] *** K4k has joined #archiveteam [19:30] *** RichardG_ has joined #archiveteam [19:34] *** gibigian1 has joined #archiveteam [19:34] *** bwn has quit IRC (Read error: Connection reset by peer) [19:34] *** Elegance has joined #archiveteam [19:34] *** Elegance has quit IRC (Connection closed) [19:34] *** bwn has joined #archiveteam [19:34] *** joepie91 has quit IRC (Ping timeout: 252 seconds) [19:35] *** Elegance has joined #archiveteam [19:37] *** joepie91 has joined #archiveteam [19:38] *** SadDM_ has joined #archiveteam [19:38] *** swebb sets mode: +o SadDM_ [19:41] *** RichardG has quit IRC (hub.se efnet.portlane.se) [19:41] *** SadDM has quit IRC (hub.se efnet.portlane.se) [19:41] *** ats has quit IRC (hub.se efnet.portlane.se) [19:41] *** Elegance_ has quit IRC (hub.se efnet.portlane.se) [19:41] *** Jordan_ has quit IRC (hub.se efnet.portlane.se) [19:41] *** gibigiana has quit IRC (hub.se efnet.portlane.se) [19:41] *** dan- has quit IRC (hub.se efnet.portlane.se) [19:41] *** diacope has quit IRC (hub.se efnet.portlane.se) [19:41] *** ats_ has joined #archiveteam [19:43] *** dan-- has joined #archiveteam [20:03] *** zer0rest has joined #archiveteam [20:05] *** K4k has quit IRC (Quit: WeeChat 1.3) [20:05] *** w0rp has quit IRC (Read error: Connection reset by peer) [20:06] *** K4k has joined #archiveteam [20:06] *** K4k has quit IRC (Connection closed) [20:06] *** K4k has joined #archiveteam [20:06] *** K4k has quit IRC (Client Quit) [20:06] *** w0rp has joined #archiveteam [20:08] *** K4k has joined #archiveteam [20:08] *** K4k has quit IRC (Remote host closed the connection!) [20:12] *** zer0rest has quit IRC (Ping timeout: 260 seconds) [20:17] *** K4k has joined #archiveteam [20:17] *** w0rp has quit IRC (Read error: Connection reset by peer) [20:20] *** bwn_ has joined #archiveteam [20:21] *** vOYtEC_ has joined #archiveteam [20:26] *** w0rp has joined #archiveteam [20:42] *** bwn has quit IRC (hub.se irc.efnet.pl) [20:44] *** Start has quit IRC (Quit: Disconnected.) [20:50] *** Start has joined #archiveteam [20:52] *** K4k_ has joined #archiveteam [20:52] *** K4k has quit IRC (Read error: Operation timed out) [21:03] *** logan has joined #archiveteam [21:03] *** nertzy2 has joined #archiveteam [21:06] *** vOYtEC_ has quit IRC (hub.efnet.us irc.Prison.NET) [21:06] *** nertzy has quit IRC (hub.efnet.us irc.Prison.NET) [21:06] *** logan2 has quit IRC (hub.efnet.us irc.Prison.NET) [21:06] *** dtm has quit IRC (hub.efnet.us irc.Prison.NET) [21:18] I asked in #archivebot as well, but I'm asking here for more coverage: Does anyone know if ArchiveBot saves videos from twitter? [21:24] *** dtm has joined #archiveteam [21:29] *** vOYtEC has joined #archiveteam [21:44] *** zer0rest has joined #archiveteam [21:46] *** brayden has joined #archiveteam [21:46] *** swebb sets mode: +o brayden [21:53] *** bentpins has joined #archiveteam [22:03] arkiver, would you happen to know the answer to my question? [22:04] hey [22:04] I'm not sure [22:04] I think so if you use youtube-dl [22:15] *** Start has quit IRC (Quit: Disconnected.) [22:28] *** K4k_ has quit IRC (Ping timeout: 260 seconds) [22:39] *** WinterFox has joined #archiveteam [22:39] *** WinterFox has quit IRC (Read error: Connection reset by peer) [22:40] *** WinterFox has joined #archiveteam [22:42] For future grabs which involve grabbing videos a check will be added to make sure BingeOn is not affecting videos if the downloader has BingeOn [22:43] I the downloader had BingeOn and it afects the videos, we will prevent the downloaders from being able to run the grab [22:44] If* has* affects* [22:45] *** megaminxw has joined #archiveteam [23:00] arkiver, I don't quite understand archive bot, could you check the dashboard and see if what is doing in nacionalrock.com is right or just wen mad? [23:06] *** BubuAnabe has quit IRC (http://www.kiwiirc.com/ - A hand crafted IRC client) [23:07] *** nickname has joined #archiveteam [23:09] I'm not sure if this relates to archiving, but I have just finished gathering all of the sha1 hashes of ISOS from WinXp to Win10 from MSDN [23:09] Here it is: http://0bin.net/paste/e6yRT8jDjdg5fUlY#Ak5BlSON4+F9ePAvRQOgc30PLJJIKrzKipo2y1LYMY6 [23:10] *** nertzy2 has quit IRC (Quit: This computer has gone to sleep) [23:11] *** vOYtEC has quit IRC (Quit: rm -r *) [23:12] *** vOYtEC has joined #archiveteam [23:12] nickname: hm, neat. feel free to dump it on the wiki in some appripriate place [23:12] This took a while, as I did manually [23:12] *did it [23:14] yo homie waz's be da secreioio code-yo [23:14] (Translation: What's the sea-creat thing for the wiki?) [23:15] yahoosucks [23:15] Thank you [23:15] more fun if you exclaim the whole forsooth line though [23:18] It is now on the wiki: http://archiveteam.org/index.php?title=Microsoft [23:21] *** zer0rest has quit IRC (Ping timeout: 260 seconds) [23:24] *** BlueMaxim has joined #archiveteam [23:25] *** bwn has joined #archiveteam [23:28] *** bwn_ has quit IRC (Read error: Operation timed out) [23:34] *** BubuAnabe has joined #archiveteam [23:36] *** BubuAnabe has quit IRC (http://www.kiwiirc.com/ - A hand crafted IRC client) [23:38] *** BubuAnabe has joined #archiveteam