[00:25] *** LS64 has joined #archiveteam [00:32] so we moved a bunch of these WARCs I uploaded to the web collection, but they aren't showing up in the wayback machine. What else is wrong? [00:32] reference https://archive.org/details/archiveteam_archivebot_go_falconk_vez_mrsk_me_20160320 which is someone's crawl of http://vez.mrsk.me [00:34] here is an example of a working one. don't see much different. https://archive.org/details/archiveteam_archivebot_go_20150504140003 [00:35] maybe a derive needs doing? I wonder how I can do that [00:37] The difference is I did one, and you did another. [00:37] If you don't upload it as type web, it confuses the thing. [00:38] I just forced a rederive [00:38] it was uploaded as type web. [00:38] is there an API way for me to rederive, or any way at all? [00:39] because I think all of the ones I've uploaded are broken. [00:42] I'm not sure. [00:45] xmc: any ideas? [00:46] hm no [00:47] there is a way to rederive, at least for me [00:47] ok, I found a way to rederive them (via the item manager) for the ones I uploaded [00:48] Hello, I'm sorry to be a bother, but I was wondering if anyone could answer a question about the Fanfiction.net archival? [00:48] hopefully there is an API, since there are quite a lot [00:50] *** logchfoo2 starts logging #archiveteam at Fri Apr 01 00:50:34 2016 [00:50] *** logchfoo2 has joined #archiveteam [00:50] would you like it to be? what's up with it? [00:50] yes it was [00:50] *** midas1 has joined #archiveteam [00:51] *** midas has quit IRC (Read error: Operation timed out) [00:51] oh, wait, I'm just bad at searching [00:51] *** lbft_ has joined #archiveteam [00:51] No, I just thought it was as it has a page, I've just been struggling to attempt to use the "getter" from github, unless it isn't supposed to work yet [00:52] *** patrickod has quit IRC (Ping timeout: 258 seconds) [00:52] *** achip has quit IRC (Ping timeout: 258 seconds) [00:52] *** lbft has quit IRC (Read error: Operation timed out) [00:52] *** Baljem_ has joined #archiveteam [00:52] *** VADemon has quit IRC (hub.efnet.us irc.Prison.NET) [00:52] *** db48x` has quit IRC (hub.efnet.us irc.Prison.NET) [00:52] *** Baljem has quit IRC (hub.efnet.us irc.Prison.NET) [00:52] lol, their robots.txt specifically disallows ia_archiver [00:53] I assume you're talking about FF, which I hate them for [00:54] such a childish decision it seems [00:54] meh [00:55] SketchCow: the rederive completed but didn't change anything [00:59] Well, if no one can really help, my apologies for bothering, I'll continue trying to download the 2012 mirror from IA [01:00] idk, does anyone have any strong feelings about crawling it? [01:01] Oh, I'm not begging for a crawl at this moment, as if I could not do it myself it'd be rude of me to ask, I was just trying to access what has already been gathered [01:01] LS64: i could help, whats your question? [01:01] I'm attempting to make use of the ffnet-grab program you point to on the page for it [01:02] LS64: story id 1-10 milliton is in archive.org/details/fanfictiondotnet_repack [01:02] LS64: fanficfare? [01:02] pip install fanficfare. much faster and easier [01:02] oh okay! [01:03] now, I downloaded the index text file from that repack prieviously [01:03] the configuration is a little tricky [01:03] I'd assume that lists every fanfiction available on there? [01:05] https://archive.org/details/fanfictiondotnet_repack has a sql file, with the metadata. 6.7 million files [01:06] Oh I didn't see the sqlite file, I'd only seen the inventory.zip [01:06] it's a recent addition. [01:07] ah, so is the inventory.zip out of date? [01:07] zip files are faster to extract a single file from than one giant tar.gz [01:08] slightly, yes, the Fanfiction_misc.zip fole contains every file that happens to start with an underscore [01:08] *** Mayeau is now known as Mayonaise [01:10] Oh okay, well, it seems like I might be outta luck, as I already searched the inventory file and didn't find what I was looking for listed, alas [01:11] LS64: what are you looking for [01:11] *** philpem has quit IRC (Ping timeout: 260 seconds) [01:12] I'm just looking for a fanfiction that I know is gone from the main site, I know it's exact name because I have a bookmark from probably 3 years ago, but it possibly was yanked before this archive was done [01:12] title? [01:12] and category? [01:14] *** BlueMaxim has quit IRC (Read error: Operation timed out) [01:15] Title... "The Secrets Behind The Rose" [01:15] *** BlueMaxim has joined #archiveteam [01:15] and sorry, as I don't quite know what you mean by "category" [01:15] LS64: what fandom? [01:15] Oh, Sonic, game fandom [01:16] I was organizing bookmarks when I found these, and once I get stuck on something, I find it hard to let go lol [01:17] fanficfare has a calibre plugin [01:17] great thing [01:17] yeah, I'm reading up on fanfifare now [01:18] anyway before we get the off-topic siren, join #archiveteam-bs [01:18] *** ariscop has quit IRC (Quit: Leaving) [01:19] *** ariscop has joined #archiveteam [01:19] *** JetBalsa has joined #archiveteam [01:20] *** useretail has joined #archiveteam [01:20] *** patrickod has joined #archiveteam [01:20] *** achip has joined #archiveteam [01:38] *** Stiletto has joined #archiveteam [01:42] *** VADemon_ has quit IRC (Quit: left4dead) [02:13] *** JesseW has joined #archiveteam [02:21] *** tomwsmf-a has joined #archiveteam [02:25] *** nwf has joined #archiveteam [02:29] *** godane has quit IRC (Quit: Leaving.) [02:41] *** ariscop has quit IRC (Quit: Leaving) [02:42] *** ariscop has joined #archiveteam [03:22] *** tomwsmf-a has quit IRC (Read error: Operation timed out) [03:54] *** metalcamp has joined #archiveteam [03:59] *** JetBalsa has quit IRC (Read error: Connection reset by peer) [04:02] *** dashcloud has quit IRC (Read error: Operation timed out) [04:10] *** dashcloud has joined #archiveteam [04:16] *** Sk1d has quit IRC (Ping timeout: 250 seconds) [04:23] *** Sk1d has joined #archiveteam [04:32] *** db48x` has joined #archiveteam [05:19] *** wp494 has quit IRC (Quit: LOUD UNNECESSARY QUIT MESSAGES) [05:28] *** wp494 has joined #archiveteam [05:51] *** WinterFox has joined #archiveteam [05:56] *** Start has quit IRC (Ping timeout: 260 seconds) [06:09] *** nwf has quit IRC (Read error: Operation timed out) [06:13] *** metalcamp has quit IRC (Ping timeout: 244 seconds) [06:17] *** LS64 has quit IRC (Ping timeout: 268 seconds) [06:24] *** Start has joined #archiveteam [06:25] *** JesseW has left [06:35] *** Deewiant_ is now known as Deewiant [06:42] *** Honno has joined #archiveteam [06:48] *** ariscop has quit IRC (Quit: Leaving) [07:29] *** schbirid has joined #archiveteam [07:36] *** vitzli has joined #archiveteam [07:36] *** Stiletto is now known as Stilett0 [07:40] *** RichardG_ has joined #archiveteam [07:41] *** RichardG has quit IRC (Ping timeout: 250 seconds) [07:45] *** RichardG has joined #archiveteam [07:47] *** RichardG_ has quit IRC (Ping timeout: 260 seconds) [07:50] *** DFJustin has quit IRC (Read error: Connection reset by peer) [07:50] *** DFJustin has joined #archiveteam [07:50] *** swebb sets mode: +o DFJustin [07:51] *** robink has quit IRC (Remote host closed the connection) [07:59] *** robink has joined #archiveteam [07:59] *** robink has quit IRC (Remote host closed the connection) [08:02] *** robink has joined #archiveteam [08:15] http://www.digitalstudies.org/ojs/index.php/digital_studies/article/view/325/412 [08:16] *** ariscop has joined #archiveteam [08:47] *** acridAxid has quit IRC (Read error: Operation timed out) [08:48] *** bwn_ has quit IRC (Read error: Operation timed out) [08:49] *** yakfish has quit IRC (Read error: Operation timed out) [08:49] *** yakfish has joined #archiveteam [08:49] *** acridAxid has joined #archiveteam [09:01] *** bwn has joined #archiveteam [09:11] *** Tomcat_ has joined #archiveteam [09:13] *** atomotic has joined #archiveteam [09:16] *** VADemon has joined #archiveteam [09:39] *** BlueMaxim has quit IRC (Quit: Leaving) [10:03] *** vitzli has quit IRC (Leaving) [10:38] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [10:57] *** Ymgve has quit IRC (Ping timeout: 506 seconds) [11:20] *** Ymgve has joined #archiveteam [11:32] *** wyatt8740 has quit IRC (Read error: Operation timed out) [11:33] *** atomotic has joined #archiveteam [11:34] *** wvdp has joined #archiveteam [11:34] *** wyatt8740 has joined #archiveteam [11:40] *** wvdp has quit IRC (http://www.kiwiirc.com/ - A hand crafted IRC client) [11:40] *** ohhdemgir has quit IRC (Read error: Connection reset by peer) [11:48] *** ohhdemgir has joined #archiveteam [11:55] *** Honno has quit IRC (Read error: Operation timed out) [12:01] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [12:16] *** Fletcher has quit IRC (Ping timeout: 244 seconds) [12:16] *** BnA-Rob1n has quit IRC (Ping timeout: 244 seconds) [12:16] *** lukeman_ has quit IRC (Ping timeout: 244 seconds) [12:17] *** Froggypwn has quit IRC (Ping timeout: 244 seconds) [12:17] *** Fletcher has joined #archiveteam [12:17] *** Froggypwn has joined #archiveteam [12:18] *** lukeman has joined #archiveteam [12:18] *** Fletcher_ sets mode: +o Fletcher [12:18] *** BnA-Rob1n has joined #archiveteam [12:27] *** bentpins has joined #archiveteam [12:33] *** MMovie has quit IRC (Read error: Operation timed out) [12:34] *** aMunster has quit IRC (Read error: Operation timed out) [12:37] *** beardicus has quit IRC (Read error: Operation timed out) [12:40] *** beardicus has joined #archiveteam [12:43] *** lbft_ has quit IRC (Read error: Operation timed out) [12:45] *** Honno has joined #archiveteam [12:45] *** lbft has joined #archiveteam [12:47] *** mhazinsk has quit IRC (Ping timeout: 634 seconds) [12:47] *** Mayonaise has quit IRC (Read error: Operation timed out) [12:49] *** Mayonaise has joined #archiveteam [12:52] *** aMunster has joined #archiveteam [13:06] *** mhazinsk has joined #archiveteam [13:07] There's a call for help in preserving whatwg (web standards group) irc logs for historian here: https://lists.w3.org/Archives/Public/public-webhistory/2016Apr/0000.html [13:09] *** godane has joined #archiveteam [13:09] *** MMovie has joined #archiveteam [13:09] tephra: http://krijnhoetmer.nl/irc-logs being archived [13:16] *** atomotic has joined #archiveteam [13:19] *** Emcy has joined #archiveteam [13:21] arkiver: great :) thank [13:21] s [13:23] So I have about $50 in DO money due to expire in a monthish. I haven't been following things with archiveteam in a while, but is any more grunt needed? Or should I spin up some TOR middle nodes [13:26] yuku [13:31] *** maseck_ has joined #archiveteam [13:31] ah cheers. I'll look at spinning up ~9 instances [13:39] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [13:40] *** davidar has quit IRC (Quit: Connection closed for inactivity) [13:43] *** maseck has quit IRC (Ping timeout: 1208 seconds) [13:44] bentpins: If you use ansible, check out the deployment script HCross and I were working on a while ago: https://github.com/HarryC145/archiveteam-deploy [13:45] *** Stilett0 is now known as Stiletto [13:46] just change the config [13:46] so the vars [13:47] Ansible plus script sounds ideal, but I'm not on ansible and I gather it's a paid service? [13:47] no it isnt [13:50] Ah great, redhat's site doesn't give that impression with "pricing" and "trials" and such. The github is more telling. I'll have a go, I'm keen to learn [13:50] cheers [13:53] *** WinterFox has quit IRC (Remote host closed the connection) [13:58] *** philpem has joined #archiveteam [14:14] *** wvdp has joined #archiveteam [14:15] *** davidar has joined #archiveteam [14:35] *** nwf has joined #archiveteam [14:42] *** atomotic has joined #archiveteam [14:58] *** Tomcat_ has quit IRC (Remote host closed the connection) [15:17] *** redlob has quit IRC (Read error: Operation timed out) [15:18] *** redlob has joined #archiveteam [15:19] *** scyther has joined #archiveteam [15:29] *** phuz has joined #archiveteam [15:30] *** phuzion has quit IRC (Read error: Connection reset by peer) [15:32] *** dserodio has quit IRC (Excess Flood) [15:32] *** RichardG has quit IRC (Ping timeout: 499 seconds) [15:34] *** dserodio has joined #archiveteam [16:04] *** RichardG has joined #archiveteam [16:18] *** JesseW has joined #archiveteam [16:29] *** RichardG has quit IRC (Ping timeout: 250 seconds) [16:30] *** phuz is now known as phuzion [16:33] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [16:48] *** JesseW has quit IRC (Ping timeout: 370 seconds) [16:51] *** Tomcat_ has joined #archiveteam [16:54] *** wvdp has quit IRC (http://www.kiwiirc.com/ - A hand crafted IRC client) [16:54] *** wvdp has joined #archiveteam [17:04] *** JetBalsa has joined #archiveteam [17:18] *** scyther has quit IRC (Quit: Leaving) [18:25] *** VADemon has quit IRC (Quit: left4dead) [18:26] *** VADemon has joined #archiveteam [18:28] *** RichardG has joined #archiveteam [18:29] *** scyther has joined #archiveteam [18:29] *** scyther has quit IRC (Connection closed) [18:36] *** Stiletto has quit IRC () [18:36] *** Stiletto has joined #archiveteam [18:37] *** RichardG has quit IRC (Ping timeout: 499 seconds) [18:43] *** RichardG has joined #archiveteam [18:47] *** bsmith093 has quit IRC (Ping timeout: 260 seconds) [19:12] *** Emcy has quit IRC (hub.se efnet.portlane.se) [19:12] *** Deewiant has quit IRC (hub.se efnet.portlane.se) [19:12] *** espes__ has quit IRC (hub.se efnet.portlane.se) [19:12] *** ats has quit IRC (hub.se efnet.portlane.se) [19:12] *** chfoo has quit IRC (hub.se efnet.portlane.se) [19:12] *** ersi has quit IRC (hub.se efnet.portlane.se) [19:12] *** goekesmi has quit IRC (hub.se efnet.portlane.se) [19:12] *** koon has quit IRC (hub.se efnet.portlane.se) [19:12] *** Fletcher_ has quit IRC (hub.se efnet.portlane.se) [19:12] *** espes___ has joined #archiveteam [19:13] *** ring has quit IRC (Read error: Operation timed out) [19:13] *** tomwsmf-a has joined #archiveteam [19:15] *** ring has joined #archiveteam [19:16] *** chfoo0 has joined #archiveteam [19:22] *** Fletcher_ has joined #archiveteam [19:22] *** ats has joined #archiveteam [19:22] *** koon has joined #archiveteam [19:22] *** Deewiant has joined #archiveteam [19:24] *** bsmith093 has joined #archiveteam [19:26] *** goekesmi_ has joined #archiveteam [19:39] *** ersi has joined #archiveteam [19:39] *** swebb sets mode: +o ersi [19:40] *** bwn has quit IRC (Read error: Operation timed out) [19:51] *** VADemon has quit IRC (Quit: left4dead) [19:58] *** bwn has joined #archiveteam [20:14] *** bwn_ has joined #archiveteam [20:23] *** bwn has quit IRC (Read error: Operation timed out) [20:44] *** Honno has quit IRC (Ping timeout: 492 seconds) [20:56] *** Tomcat_ has quit IRC (Remote host closed the connection) [20:59] *** ats has quit IRC (Ping timeout: 250 seconds) [21:00] *** BlueMaxim has joined #archiveteam [21:00] *** ats has joined #archiveteam [21:00] *** ersi has quit IRC (Ping timeout: 250 seconds) [21:01] *** Deewiant_ has joined #archiveteam [21:08] *** Fletcher_ has quit IRC (hub.se efnet.portlane.se) [21:08] *** koon has quit IRC (hub.se efnet.portlane.se) [21:08] *** Deewiant has quit IRC (hub.se efnet.portlane.se) [21:15] *** vtyl has quit IRC (Ping timeout: 260 seconds) [21:17] *** BlueMaxim has quit IRC (Read error: Operation timed out) [21:18] *** vtyl has joined #archiveteam [21:19] *** bwn_ is now known as bwn [21:24] *** chfoo0 is now known as chfoo [21:33] *** bwn has quit IRC (Read error: Operation timed out) [21:34] *** bwn has joined #archiveteam [21:46] *** WinterFox has joined #archiveteam [21:49] *** Deewiant_ has quit IRC (hub.se irc.du.se) [21:49] *** Sk1d has quit IRC (hub.se irc.du.se) [21:56] *** bwn has quit IRC (Leaving) [21:57] *** bwn has joined #archiveteam [22:18] *** db48x` is now known as db48x [22:24] *** schbirid has quit IRC (Quit: Leaving) [22:24] *** Boppen has joined #archiveteam [22:29] *** wvdp has quit IRC (http://www.kiwiirc.com/ - A hand crafted IRC client) [22:37] *** aliz has quit IRC (Remote host closed the connection) [23:09] *** midas1 is now known as midas [23:24] *** tubenerd1 has joined #archiveteam [23:24] *** Microguru has quit IRC (Read error: Connection reset by peer) [23:25] archive team, you available? [23:25] *** tubenerd1 has quit IRC (Client Quit) [23:26] *** nillary16 has joined #archiveteam [23:27] *** nillary16 has quit IRC (Client Quit) [23:36] *** tomwsmf-a has quit IRC (Read error: Operation timed out) [23:52] sure- what's up? [23:55] *** JesseW has joined #archiveteam [23:59] I think he left