[00:09] *** bzc6p_ has joined #archiveteam [00:12] *** bwn has quit IRC (Read error: Operation timed out) [00:16] *** bzc6p has quit IRC (Ping timeout: 615 seconds) [00:22] *** Ravenloft has joined #archiveteam [00:24] *** aaaaaaaaa has quit IRC (Read error: Connection reset by peer) [00:25] *** aaaaaaaaa has joined #archiveteam [00:30] *** Meeh has quit IRC (Read error: Operation timed out) [00:30] *** Meeh has joined #archiveteam [00:38] *** wednesday has quit IRC (Read error: Operation timed out) [00:38] *** wednesday has joined #archiveteam [00:51] *** Elegance_ has joined #archiveteam [00:52] *** Elegance has quit IRC (Read error: Connection reset by peer) [01:14] *** JesseW has joined #archiveteam [01:20] *** xmc has quit IRC (Read error: Operation timed out) [01:21] *** slyphic|a has quit IRC (Read error: Operation timed out) [01:21] *** ats has quit IRC (Read error: Operation timed out) [01:22] *** Elegance_ has quit IRC (Read error: Operation timed out) [01:22] *** mistym has quit IRC (Ping timeout: 369 seconds) [01:22] *** xmc has joined #archiveteam [01:23] *** Fusl has quit IRC (Ping timeout: 255 seconds) [01:23] *** Elegance has joined #archiveteam [01:23] *** HCross has quit IRC (Read error: Operation timed out) [01:23] *** mistym has joined #archiveteam [01:24] *** nico_32 has quit IRC (Ping timeout: 369 seconds) [01:25] *** chazchaz has quit IRC (Ping timeout: 369 seconds) [01:26] *** dcmorton has quit IRC (Ping timeout: 369 seconds) [01:26] *** dxrt has quit IRC (Ping timeout: 369 seconds) [01:27] *** ats has joined #archiveteam [01:27] *** phuzion has quit IRC (Ping timeout: 369 seconds) [01:27] *** atlogbot has quit IRC (Ping timeout: 369 seconds) [01:27] *** phuzion has joined #archiveteam [01:27] *** ironman_ has quit IRC (Ping timeout: 255 seconds) [01:27] *** wacky has quit IRC (Ping timeout: 369 seconds) [01:27] *** JesseW has quit IRC (Leaving.) [01:29] *** Fusl has joined #archiveteam [01:29] *** nico_32 has joined #archiveteam [01:29] *** dxrt has joined #archiveteam [01:30] *** ironman_ has joined #archiveteam [01:30] *** atlogbot has joined #archiveteam [01:31] *** chazchaz has joined #archiveteam [01:31] *** dcmorton has joined #archiveteam [01:33] *** HarryCros has joined #archiveteam [01:33] *** wacky has joined #archiveteam [01:35] *** Ravenloft has quit IRC (Ping timeout: 360 seconds) [01:39] *** remsen2 has joined #archiveteam [01:39] *** remsen2 has quit IRC (Remote host closed the connection) [01:44] *** remsen has quit IRC (Read error: Operation timed out) [01:48] *** slyphic has joined #archiveteam [01:50] *** bwn has joined #archiveteam [02:18] *** Ungstein has joined #archiveteam [02:24] *** balrog has quit IRC (Read error: Connection reset by peer) [02:40] *** aaaaaaaaa has quit IRC (Read error: Operation timed out) [02:46] *** balrog has joined #archiveteam [02:48] arkiver: The situation in Paris has quieted down so it's probably safe to resume GameFront [02:49] *** balrog has quit IRC (Read error: Connection reset by peer) [03:03] *** Meeh has quit IRC (Read error: Operation timed out) [03:04] *** Meeh has joined #archiveteam [03:11] *** primus104 has quit IRC (Leaving.) [03:47] *** balrog has joined #archiveteam [03:48] *** xk_id_ has quit IRC (Remote host closed the connection) [03:55] *** bwn has quit IRC (Read error: Operation timed out) [04:15] *** bwn has joined #archiveteam [04:37] *** Ymgve has quit IRC () [05:00] *** JesseW has joined #archiveteam [05:34] *** RedType_ has joined #archiveteam [05:36] *** RedType has quit IRC (Read error: Operation timed out) [05:39] What's the plan for ADrive? Their deadline is this monday; the channel is #bdrive -- does anyone know the current status? [05:39] Are we going to make a warrior, or toss URLs in archivebot, or? [05:54] *** dashcloud has quit IRC (Read error: Operation timed out) [05:58] *** dashcloud has joined #archiveteam [06:01] *** bwn has quit IRC (Read error: Operation timed out) [06:11] *** GLaDOS has quit IRC (Read error: Operation timed out) [06:13] *** remsen has joined #archiveteam [06:22] *** bwn has joined #archiveteam [06:34] SketchCow: https://archive.org/details/KlikAndPlayVideos [06:41] *** JesseW has quit IRC (Leaving.) [06:43] *** GLaDOS has joined #archiveteam [06:58] *** Ungstein has quit IRC (Quit: Leaving.) [07:04] *** Ungstein has joined #archiveteam [07:28] *** Ungstein has quit IRC (Quit: Leaving.) [07:48] *** Ungstein has joined #archiveteam [08:00] *** Ungstein has quit IRC (Read error: Connection reset by peer) [08:04] *** Ungstein has joined #archiveteam [08:33] *** nertzy has joined #archiveteam [08:40] *** WinterFox has quit IRC (Remote host closed the connection) [08:42] *** WinterFox has joined #archiveteam [08:57] *** dashcloud has quit IRC (Read error: Operation timed out) [09:00] *** dashcloud has joined #archiveteam [09:16] *** primus104 has joined #archiveteam [09:27] *** nertzy has quit IRC (Quit: This computer has gone to sleep) [09:27] *** jmad980 has quit IRC (Ping timeout: 360 seconds) [09:37] *** jmad980 has joined #archiveteam [09:44] *** primus104 has quit IRC (Leaving.) [10:00] *** BlueMaxim has quit IRC (Quit: Leaving) [10:09] *** schbirid has joined #archiveteam [10:16] *** primus104 has joined #archiveteam [10:27] *** VADemon has quit IRC (left4dead) [10:51] *** aMunster has quit IRC (Read error: Operation timed out) [10:53] *** dxrt has quit IRC (Read error: Operation timed out) [10:55] *** aMunster has joined #archiveteam [10:55] *** dxrt has joined #archiveteam [11:02] videos about Paris appear to be getting deleted or hidden from youtube [11:03] I'm running a periodic ytdl of all search results for 'Paris' [11:03] and it was 5 pages worth of relevant results a few hours ago [11:03] and it's still 5 pages worth of relevant results now [11:03] but things have been added in the meantime [11:03] so it doesn't add up [11:03] not really sure what's going on there [11:06] *** WubTheCap has joined #archiveteam [11:06] SketchCow: https://catalogd.archive.org/log/441029088 <- it appears a primary just went poof? [11:06] I had two S3 uploads fail with this [11:06] via HTML5 uploader [11:12] also, looks like demonii is dead [11:12] *** aMunster has quit IRC (Read error: Operation timed out) [11:13] *** dxrt has quit IRC (Read error: Operation timed out) [11:21] *** dxrt has joined #archiveteam [11:22] *** aMunster has joined #archiveteam [11:24] Madokami (a Pomf clone) went down. Maintainer stopped paying server bills, claims it being a financial burden. All files are currently inaccessible, but qqueue is taking over. [11:25] If it comes back up, I guess there's another task for a manual WARC or a warrior project [11:25] https://git.pantsu.cat/WubTheCaptain/deathwatch-pomf#madokamicom [11:27] *** dashcloud has quit IRC (Read error: Operation timed out) [11:27] JesseW: see what I wrote a in #bdrive [11:28] joepie91: happens sometimes, just restart the failed process and it'll work [11:28] arkiver: SketchCow: so I tried that, twice for one item, and now I have two queued tasks, but the uploader said it was an error, but the JSON response from the API said all was fine [11:28] https://catalogd.archive.org/history/TheNordenS01E03-Religion [11:29] this looks like it needs some admin attention :P [11:29] idem for https://catalogd.archive.org/history/TheNordenPoliceS01E06-Police (which I only retried once) [11:29] I reran 03, working now [11:30] 06 too [11:30] arkiver: do the extra tasks not need cancelling then? [11:30] ah [11:30] right :P [11:30] I think those will be fine [11:30] (why can't I rerun tasks D: ) [11:31] *** dashcloud has joined #archiveteam [11:31] arkiver: yeah, they just went poof [11:31] from the list [11:31] yes [11:31] mmk, looks all good now [11:31] * joepie91 rm originals [11:31] house cleaning day.. [11:31] well, HDD cleaning [11:42] *** VADemon has joined #archiveteam [11:43] *** primus104 has quit IRC (Leaving.) [11:49] *** bwn has quit IRC (Read error: Operation timed out) [11:53] *** bzc6p_ is now known as bzc6p [11:56] joepie91: Hardsubbed. Why. [11:56] *** its_notja has joined #archiveteam [11:56] *** its_notja is now known as notjack [11:57] WubTheCap: I don't know. hence the upload [11:57] WubTheCap: I suspect it's some sort of Chinese bootleh [11:57] bootleg * [11:57] so it might have other oddities in it [11:57] which makes it interesting ;) [11:58] *** HarryCros is now known as HCross [12:05] I've just read through WubTheCap's exhaustive list of pomf.se clones (thanks for that). If only those who kick off with such services, would be more serious and would think it over! [12:06] "I'm busy with my stuff... I think I'll burn the service down to the ground, what else could I do?" Please. [12:07] Linkrot ftw. [12:09] joepie91: thanks for the Paris stuff. [12:10] I haven't seen botpie around recently. [12:10] *** bzc6p has left [12:10] *** bzc6p has joined #archiveteam [12:13] * joepie91 cough [12:13] *** dashcloud has quit IRC (Read error: Operation timed out) [12:14] *** botpie91 has joined #archiveteam [12:14] bai: sure you have :P [12:14] er [12:14] bzc6p: * [12:14] hm [12:14] .ping [12:14] Ping failed. Are you sure you specified a valid hostname or IP? [12:14] weird [12:14] I accidentally my SSH session [12:14] bzc6p: At least tfwno.space quit only a day or two later once the maintainer figured out it's not serious and ready yet [12:15] Fuwa.se is the only "serious" clone out there [12:15] Very approachable maintainer too [12:16] 1339.cf is also okay [12:16] Others are meh [12:16] WubTheCap: I missed the few days part. Well, then... [12:16] okay but not great* [12:16] These sites seriously need a business model [12:16] *** dashcloud has joined #archiveteam [12:17] I could sponsor few, like I sponsored Pomf.se briefly before shutdown [12:17] And it's not about "business model" really [12:18] If one wants the service earn itself the expenses, should put there ads or offer a premium plan. [12:18] I've told myself you either 1. be for-profit and make a full blown jew service that upsets the community or 2. be non-profit and shutdown someday in future. Nobody has tasked to do the latter with a real registered organization, only individuals. [12:18] If one doesn't want to run out of space, then limit the filesize and/or delete old/unaccessed files. [12:18] bzc6p: https://drewdevault.com/2014/10/10/The-profitability-of-online-services.html [12:19] I can make these pomf clones into a warrior project if we have lists of URLs [12:19] If one wants to keep dmca away, shouldn't let files be multiple hundreds of megabytes in size. And should have some censoring. [12:19] It's not really the amount of files, the bandwidth is the problem (hotlinking driving bandwidth per month to 150TB+) [12:19] Just a few things to make such a service serious. [12:19] supposedly, ISIS claimed the attack [12:19] I can't find the video [12:19] nobody is linking to it [12:19] is it on IA already? [12:20] arkiver: The ones we have URLs for are already archived from about last month. Fuwa.se is a special case, it archives every month. [12:20] The ones we don't have URLs for, well, they don't want to share the file lists. [12:20] are they archived as WARCs? [12:20] arkiver: Yes. [12:20] ok [12:20] WubTheCap: wikipedia is non-profit, IA is non-profit [12:20] also, bandwidth costs fuck all [12:20] if you're going bankrupt on bandwidth, you're using the wrong providers [12:21] (and indeed, many of these sites do) [12:21] joepie91: I suppose, but they also have much more funding (and grants?). In Finland, I can't even run a fundraiser because it requires a permit from the police department to do it legally [12:21] WubTheCap: doesn't change that non-profit is very possible [12:21] just requires thought [12:21] From what I've queried, people are mostly only ready to pay 12 or 24 EUR per year (that's 1-2 EUR/month) for supporting a Pomf clone or an organization [12:22] It's possible but hard [12:22] You need tons of people to be existing members in an organization [12:22] WubTheCap: so, with 10 subscribers, you have the operating costs of a small image host covered [12:22] but, -bs :) [12:22] joepie91: ? [12:22] I said per year. [12:22] WubTheCap: yes [12:23] WubTheCap: join #archiveteam-bs [12:23] I calculated it to be more like 30+ [12:23] for further discussion [12:23] this is a strictly on-topic/brevity channel ;) [12:23] *** bzc6p_ has joined #archiveteam [12:26] *** bzc6p has quit IRC (Read error: Operation timed out) [12:28] I've updated the 'Froogle' page's site and archive status as per the archiveteam pages and my personal research. [12:28] *statuses [12:32] Edited Club Nintendo to reflect the fact it is now offline. [12:42] *** bwn has joined #archiveteam [12:46] *** nertzy has joined #archiveteam [12:47] *** antomatic has quit IRC (Read error: Operation timed out) [13:10] *** antomatic has joined #archiveteam [13:14] *** chazchaz has quit IRC (Read error: Operation timed out) [13:18] *** nertzy has quit IRC (Quit: This computer has gone to sleep) [13:20] *** chazchaz has joined #archiveteam [13:21] *** Ungstein has quit IRC (Quit: Leaving.) [13:25] *** RichardG has quit IRC (Read error: Connection reset by peer) [13:27] *** RichardG has joined #archiveteam [13:32] *** remsen has quit IRC (Read error: Connection reset by peer) [13:37] is there no way to upload to an IA item while ignoring things that are already there? [13:38] *** bzc6p_ is now known as bzc6p [13:39] joepie91: what do you mean by ignoring? [13:39] my browser crashed mid-upload [13:39] I want to skip the ones already there, preferably using terminal script [13:39] ie. internetarchive module [13:39] ias3upload ignores [13:40] I have no idea how to add files with ias3upload [13:40] you have to create a csv [13:40] *** Ymgve has joined #archiveteam [13:41] Probably a bit more of micromanagement, but a bash script could be written to handle that. [13:41] internetarchive doesn't ignore? [13:41] apparently not. [13:49] ls | sort | grep -v \.log > local.log [13:50] ia list -c name YouTube-Nopefully | sort > already-uploaded.log [13:50] comm --nocheck-order -23 local.log already-uploaded.log | xargs -d '\n' ia upload --log YouTube-Nopefully [13:50] because, y'know, let's keep things simple [13:50] lol [13:50] @ bzc6p [13:51] *** antomatic has quit IRC (Read error: Operation timed out) [13:52] I love it when I have a problem, I take bash, and then I don't have the problem anymore. It needs to be written only once. And it's quite powerful.- [13:52] *** Ravenloft has joined #archiveteam [13:52] bzc6p: and then you find out halfway through that fuck a stray apostrophe screwed up half your uploads and now you have two problems, both of which are poorly documented [13:52] :) [13:53] three, actually [13:53] your original problem [13:53] the broken bit in your bash script [13:53] and how to fix what you screwed up [13:53] Still, it's worth the time. [13:55] *** WinterFox has quit IRC (Remote host closed the connection) [13:57] *** antomatic has joined #archiveteam [14:04] Also, I usually output the results first to make sure it works as expected, before actually having the script do the stuff. [14:08] Is robots.txt still bad if I exclude the internet archive bot? [14:10] notjack: what do you intend to avoid by using robots.txt? [14:10] @bzc6p I have a page that I'm hosting for a friend that is completely unrelated to the website, and it is appearing on google. [14:10] I don't mind it being archived, but I'd rather not have it on google [14:15] I thought to implement respecting robots.txt to Fuuka (4chan archiver) with override option [14:25] *** primus104 has joined #archiveteam [14:31] *** xk_id has joined #archiveteam [14:34] joepie91: Were you looking for this? http://www.liveleak.com/view?i=8c7_1447505244 [14:35] notjack: sounds good [14:36] WubTheCap: danke [14:36] (it was indeed) [15:00] *** schbirid has quit IRC (Quit: Leaving) [15:04] *** Froggypwn has quit IRC (Read error: Connection reset by peer) [15:04] *** xk_id has quit IRC (Remote host closed the connection) [15:05] *** xk_id has joined #archiveteam [15:05] *** Froggypwn has joined #archiveteam [15:22] *** xk_id has quit IRC (Read error: Operation timed out) [15:27] *** Stiletto is now known as Stilett0 [15:38] *** primus104 has quit IRC (Leaving.) [15:44] *** antomatic has quit IRC (Ping timeout: 258 seconds) [15:49] *** Administr has joined #archiveteam [15:51] *** antomatic has joined #archiveteam [15:52] *** ironman_ has quit IRC (Ping timeout: 255 seconds) [15:52] *** ironman_ has joined #archiveteam [15:53] *** HarryCros has joined #archiveteam [15:54] *** jmad980 has quit IRC (Read error: Operation timed out) [15:54] *** HCross has quit IRC (Read error: Operation timed out) [15:54] *** ex-parrot has quit IRC (Read error: Operation timed out) [15:54] *** Famicoman has quit IRC (Read error: Operation timed out) [15:54] *** antomatic has quit IRC (Read error: Operation timed out) [15:55] *** Administr has quit IRC (Read error: Operation timed out) [15:55] *** antomatic has joined #archiveteam [15:55] *** cadbury has quit IRC (Read error: Operation timed out) [15:56] *** jmad980 has joined #archiveteam [15:56] *** cadbury has joined #archiveteam [15:56] *** JW_work has quit IRC (Read error: Connection reset by peer) [15:58] *** JW_work has joined #archiveteam [15:58] *** JW_work has quit IRC (Read error: Connection reset by peer) [15:58] *** antomatic has quit IRC (Read error: Operation timed out) [15:59] *** xk_id has joined #archiveteam [16:00] *** ex-parrot has joined #archiveteam [16:01] *** antomatic has joined #archiveteam [16:02] *** Elegance has quit IRC (Read error: Operation timed out) [16:02] *** JW_work has joined #archiveteam [16:03] *** Famicoman has joined #archiveteam [16:03] *** Elegance has joined #archiveteam [16:11] *** xk_id has quit IRC (Remote host closed the connection) [16:13] *** xk_id has joined #archiveteam [16:32] *** scyther has joined #archiveteam [16:47] *** primus104 has joined #archiveteam [17:03] *** notjack has quit IRC (Ping timeout: 240 seconds) [17:13] *** notjack has joined #archiveteam [17:24] *** maseck has quit IRC (Remote host closed the connection) [17:30] *** Ravenloft has quit IRC (Ping timeout: 360 seconds) [17:30] *** xk_id has quit IRC (Remote host closed the connection) [17:30] *** xk_id has joined #archiveteam [17:37] *** maseck has joined #archiveteam [17:41] *** xk_id has quit IRC (Read error: Operation timed out) [17:51] *** JesseW has joined #archiveteam [18:01] *** philpem has quit IRC (Ping timeout: 252 seconds) [18:07] *** philpem has joined #archiveteam [18:11] *** maseck has quit IRC (Remote host closed the connection) [18:14] *** maseck has joined #archiveteam [18:25] *** xk_id has joined #archiveteam [18:38] *** bwn_ has joined #archiveteam [18:39] *** JesseW has quit IRC (Leaving.) [18:46] *** bwn has quit IRC (Read error: Operation timed out) [18:47] *** pikhq has quit IRC (Remote host closed the connection) [18:49] *** pikhq has joined #archiveteam [18:51] *** HarryCros is now known as HCross [18:53] chfoo: can you please create a rsync target on FOS for adrive? [18:57] *** aaaaaaaaa has joined #archiveteam [19:09] *** maseck_ has joined #archiveteam [19:12] *** maseck has quit IRC (Read error: Operation timed out) [19:14] *** maseck_ has quit IRC (Client Quit) [19:15] *** maseck has joined #archiveteam [19:16] *** xk_id has quit IRC (Remote host closed the connection) [19:32] *** JesseW has joined #archiveteam [19:34] *** maseck has quit IRC (Remote host closed the connection) [19:46] *** maseck has joined #archiveteam [20:22] *** arkiver2 has joined #archiveteam [20:27] *** arkiver2 has quit IRC (Ping timeout: 252 seconds) [20:33] *** xk_id has joined #archiveteam [20:35] *** xk_id_ has joined #archiveteam [20:35] *** xk_id has quit IRC (Read error: Connection reset by peer) [20:44] *** VADemon has quit IRC (left4dead) [20:52] *** WinterFox has joined #archiveteam [20:53] anybody with access to FOS able to set up a rsync target for #bdrive? (apologies if this has already been asked) [21:02] oh no we're running out of time [21:36] *** notjack has quit IRC (Ping timeout: 242 seconds) [21:46] *** bwn_ has quit IRC (Ping timeout: 606 seconds) [21:55] *** scyther has quit IRC (Read error: Connection reset by peer) [22:13] *** bwn has joined #archiveteam [22:43] *** mattis_ has joined #archiveteam [23:07] *** Rickster has joined #archiveteam [23:08] *** mattis_ has quit IRC (Ping timeout: 240 seconds) [23:20] *** JesseW has quit IRC (Leaving.) [23:24] *** raven_ has joined #archiveteam [23:34] *** Rickster has quit IRC (Quit: ZNC - http://znc.in) [23:37] *** Rickster has joined #archiveteam [23:48] we need an rsync target for adrive as soon as possible [23:49] *** BlueMaxim has joined #archiveteam [23:49] it should have good speed [23:49] we don't know how large the grab will be, but to be save it should have at least 1 TB of space [23:49] #bdrive [23:51] ping midas achip [23:52] Anyone who is an expert at setting up rsync destinations? [23:57] I know rsync [23:59] Sorry miss read