[00:06] *** MMovie has quit IRC (Read error: Operation timed out) [00:07] *** MMovie has joined #archiveteam [00:22] *** Stilett0 is now known as Stiletto [00:35] *** redlob has quit IRC (Read error: Operation timed out) [00:37] *** dashcloud has quit IRC (Read error: Operation timed out) [00:39] *** dashcloud has joined #archiveteam [00:45] *** redlob has joined #archiveteam [00:53] *** MMovie has quit IRC (Read error: Operation timed out) [00:55] *** MMovie has joined #archiveteam [01:08] *** MMovie has quit IRC (Read error: Operation timed out) [01:09] *** MMovie has joined #archiveteam [01:11] *** JesseW has joined #archiveteam [01:12] so FOS is still looking really congested [01:12] like 50-100kB/s even after connections were limited to 50 concurrent [01:12] is there some way I can configure it to assemble ~25GB megawarcs and upload more directly to IA? [01:22] yeah, you'll want the archiveteam-megawarc-factory git repo, an IA account, IA-S3 keys (https://archive.org/account/s3.php) and ideally access to the collection you want to put stuff in [01:22] I can assist with setup of the whole smash except for the last bit, though I am booked until tomorrow [01:23] I think there must be others who know how it works, though -- xmc chfoo ersi joepie91 maybe [01:23] or they can be dereferenced [01:23] yes hi [01:23] yeah xmc is to my left. [01:23] *** n00b599 has joined #archiveteam [01:23] *** dashcloud has quit IRC (Read error: Operation timed out) [01:23] wait, physically [01:23] r/ring secret word. [01:23] * xmc waves [01:23] I can do the software thing but the account and keys and access I obviously need help with :) [01:24] not super urgent - tomorrow is just fine [01:24] *** dashcloud has joined #archiveteam [01:24] IA account is just any IA account, keys can be generated at that link [01:24] the access thing, ping SketchCow [01:24] * FalconK pings at SketchCow [01:24] Didn't work. [01:25] Oh got it [01:25] yahoosucks [01:26] Thank ya [01:27] sure thing [01:27] what're you up to? [01:29] *** Anon has joined #archiveteam [01:29] *** Anon has quit IRC (Client Quit) [01:30] *** Anon has joined #archiveteam [01:31] loveisover is down. [01:31] *** Anon has quit IRC (Client Quit) [01:40] *** philpem has quit IRC (Ping timeout: 260 seconds) [01:47] can I not just upload things to the archivebot collection? [01:47] perhaps I will try that [01:49] i think that perhaps we should consider giving each pipeline operator a sub-collection inside of archivebot, or maybe just privileges to upload into it [01:49] hm [01:49] i'm not sure [01:49] there are many ways that we could do this differently [01:53] oh [01:53] archivebot collection [01:53] yeah I think SketchCow's the only one who has that [01:53] * yipdw doesn't have that access [01:54] yeah [01:54] sounds about right [01:54] we've been talking about this for a little while and i still can't think of a good reason for archivebot to depend on fos [02:01] Oh were you talking to me at 19:27? [02:02] it's not even 19:00, friend [02:03] Central time zone best time zone [02:03] * MrRadar puts on shades [02:04] YEEEEEEEEEEEEEEEEEEEAAAAAAAAAAAAAAAAAH!!! [02:05] I'm from Miami for real, so I guess you can say that meme comes with the nativety. [02:06] It's arguably as good or better than Ocarina of Time in some ways. In some ways it's not. I beat it 10 years ago on GCN and don't remember any of it so I've been playing it on the 3DS and it's awesome. I've died and reset the game from the beginning twice in the Water Temple already (the first time I ran out of oxygen in a room where I couldn't figure out what to do and the second time I got killed by Stalfos) so I have [02:06] woop woop woop off-topic siren [02:06] Arguabely, it's better to have a single collection point, so you aren't dealing with maintaining multiple systems [02:06] I was testing how the IRC formatted.. That's what I was typing out when you asked what I was up to. [02:07] ErkDog: well it's overloaded and slowing down archivebot [02:07] Do you guys 4chan? [02:08] why do you ask? [02:08] That's what I signed up here for. That and I'm an archivist in my own right. I'm surprised I never heard of this site. [02:08] are you ANOTHER 4chan archiver? [02:08] I [02:08] I [02:08] I'm looking for the data from May to October 4th. [02:09] Negatory. [02:09] for what [02:09] Just a frequent poster who had a lot of posts I wanted to reread. [02:09] beats me. we don't archive 4chan actively. [02:09] Hisotircal preservation on the note as well. [02:10] Ah, I see. What I'm referring to is a good Samaritan that I read about on the 4chanarchive page who had a private archive and was volunteering to share it. [02:10] no, this is not an archive warez channel, we don't keep a meat-index of stuff [02:10] all I can say is http://archive.org/search.php [02:11] Oh that's what you're affiliated directly with? [02:11] no [02:11] We use the IA as a repository for our work but there is no official connection [02:11] but things wind up there [02:11] Gotcha [02:12] If you know someone with archived data to share, ask him to upload it there [02:12] That way it will be preserved [02:12] Well. if nobody's heard what I'm talking about, I think I found what I need. I'll just make a section on the discussion page/ [02:12] *nod*nod* [02:24] *** lokis has joined #archiveteam [02:24] o/ [02:25] *** MMovie has quit IRC (Read error: Operation timed out) [02:26] *** MMovie has joined #archiveteam [02:27] yeah it is XMC, lol [02:28] I mentioned previously that it might be a good idea to have reginal RSYnc systems so like for instance, I could pick a target in the US [02:28] EU people could pick EU [02:29] then those targets could single threaded push stuff over to FOS so it wasn't gettings it's IO thrashed by 50 simultaneous RSYNCHs coming in [02:29] but instead 2 or 3 [02:29] and then the back end could keep up [02:29] there is no point in using rsync for this at all [02:29] since it looks like FOS is merely forwarding the archives as-is into IA collection archivebot [02:29] just send them directly using the HTTP REST API [02:30] i don't think ia supports giving multiple users access to a restricted collection? [02:30] like bouncing them through FOS is a waste of bandwidth [02:30] not really because archive bot is on the same network [02:30] but subcollections, one per archivebot pipeline owner, make sense [02:30] have you ever tracerouted FOS? [02:30] yes, it's in IA [02:30] that is fine [02:30] it's clearly overloaded, and a bottleneck, and it doesn't need to be so it shou;dn't be [02:31] we're not saturating IA downstream [02:31] or our upstream [02:31] yes, but that's because FOS is dealing with 50 incoming rsynch's [02:31] point of order, i would say "inbound" and "outbound" [02:31] instead of 2 or 3 from front end systems [02:31] so it doesn't matter why [02:31] the bottleneck is in FOS's disk bandwidth [02:31] there is just literally no reason to have that extra infrastructure [02:32] well if Archive Bot could pull the data from the other RSynch servers, then sure [02:32] why use rsync at all? [02:32] because rsync does CRC checking [02:32] HTTP and standard FTP don't [02:33] the BER is probably the same, since FOS uploads over a network too [02:33] also doesn't WARC have a CRC? [02:34] I wouldn't call a transfer inside the same building a "network" upload that's succeptible to the same potential for corruption as sending data across the open internet [02:34] I'm not sure if WARC does or not, but if it did, you'd have to do the CRC after transmit, and then retransmit the entire WARC file if it failed, rsync does the CRC in real time as things are transmitted [02:34] why do you h8 rsync? did it murder your family? [02:36] TCP has a CRC [02:36] on every packet! [02:36] And yes WARC files have checksums on each record too [02:36] it's part of the error control mechanism of the internet at large [02:36] it works pretty well [02:36] I got a bad vibe. [02:36] i've never received an accidentally corrupted tcp stream [02:37] ohhh good point TCP does [02:37] What;s a TCP screen? [02:37] so [02:37] if someone wants to write a different uploader for ArchiveBot, I am down with that [02:37] however, that doesn't change the fact, that I've personally seen, and experienced FTP and HTTP transfers, using TCP which ended up corrupted and required re-transmittal [02:37] I am going to, yipdw :) [02:37] already in progress :) [02:37] ideally, it would just require replacement of uploader.py [02:38] yup [02:38] there. blessings from two people with @ and i'm sure SketchCow will be happy to get it off FOS [02:38] *** marvinw_ is now known as ivan` [02:38] it would however require each operator to have upload privileges into the archivebot collection, as well as a way to generate item names [02:38] is why i propose a subcollection for operators [02:39] though that would mean that whoever is in charge of archivebot have permissions to create these collections [02:39] my knowledge of IA's permissions mechanisms is, sadly, lacking [02:39] anyway we'll get the tool first and figure out access later [02:39] * xmc nods [02:39] and are people seriously talking about TCP checksums and corrupted data w tf [02:40] * yipdw alt+tab [02:40] lol yipdw [02:40] ALSO [02:40] having the pipeline operator upload directly actually gives us a much better audit trail [02:40] which is a thing i've been thinking about for a little while [02:41] Well without changing the existing access permissions, having regional front end rsynch targets which then sent single threaded to FOS would lessen it's disk I/O significantly. [02:41] 50 RSynchs run more than 50 times slower than 1, even 2 or 3 [02:41] why are you so stuck on rsync [02:41] well cause that's just how it works now [02:41] so [02:41] we've seen this behavior, yes [02:41] and would be easier to change the work flow [02:41] it works that way now because when we wrote it we were feeling lazy [02:42] than having IA give access permissions that they may or may not be willing to do [02:42] sometimes it is the correct time to redesign things [02:42] now is, apparently, the correct time [02:42] SketchCow can sort out any IA permissions we need. It's not an issue [02:42] well that and if you give each operator subcollections the problem goes away, ish [02:42] OMG, if I could submit work loads at faster than 100K/sec I would be immensely happy :-D [02:42] yes [02:43] we share a common goal :D [02:43] this isn't an invitation to shove fucking en.wikipedia.org into archivebot [02:43] !a en.wikipedia.org [02:43] er [02:43] :P [02:43] you need http:// [02:43] I think you can just download their whole SQL database anyway. no reason to crawl it. [02:43] that's what WIKITEAM is for [02:43] i think Nemo_bis is in charge of that [02:44] come to think of it, ia upload plus shell script is about all you need [02:44] well [02:44] and a 64-bit system [02:44] ha [02:45] all that and a bag of chips [02:45] I mean curl or whatever, it doesn't matter really [02:52] *** dashcloud has quit IRC (Read error: Operation timed out) [02:53] I' [02:54] *** mismatch_ has quit IRC (Remote host closed the connection) [02:54] once we get permissions set up I can test my thing [02:54] *** mismatch_ has joined #archiveteam [02:56] *** dashcloud has joined #archiveteam [03:09] *** MMovie has quit IRC (Read error: Operation timed out) [03:10] *** MMovie has joined #archiveteam [03:11] *** n00b599 has quit IRC (Quit: Page closed) [03:22] *** superkuh has quit IRC (Quit: the neuronal action potential is an electrical manipulation of reversible abrupt phase changes in the lipid bilaye) [03:23] *** MMovie has quit IRC (Read error: Operation timed out) [03:24] *** MMovie has joined #archiveteam [03:37] *** bwn has quit IRC (Read error: Operation timed out) [03:37] *** MMovie has quit IRC (Read error: Operation timed out) [03:38] *** MMovie has joined #archiveteam [03:40] *** Stiletto has quit IRC (Read error: Connection reset by peer) [03:41] *** Stolett0 has joined #archiveteam [03:44] SketchCow: BTW, when I try to turn on sound on https://archive.org/details/msdos_Alpine_Tram_Ride_1989 it tells me: "This button only works once the emulation is running" even after the emulator appears to be running. IDK if I should email this to you, to info, both or neither. [03:44] *** Stolett0 is now known as Stiletto [03:48] *** Stolett0 has joined #archiveteam [03:58] *** Stiletto has quit IRC (Read error: Operation timed out) [03:59] *** vitzli has joined #archiveteam [03:59] *** MMovie has quit IRC (Read error: Operation timed out) [04:00] *** Stolett0 has quit IRC (Read error: Connection reset by peer) [04:00] *** MMovie has joined #archiveteam [04:01] *** Stolett0 has joined #archiveteam [04:17] *** MMovie has quit IRC (Read error: Operation timed out) [04:19] *** MMovie has joined #archiveteam [04:25] *** Stolett0 is now known as Stiletto [04:26] *** Stiletto is now known as Stilett0 [04:26] *** Stilett0 is now known as Stiletto [04:29] *** tomwsmf-a has quit IRC (Read error: Operation timed out) [04:44] *** MMovie has quit IRC (Read error: Operation timed out) [04:45] *** MMovie has joined #archiveteam [05:02] *** MMovie has quit IRC (Read error: Operation timed out) [05:04] *** MMovie has joined #archiveteam [05:07] *** RedType has joined #archiveteam [05:07] *** RichardG has quit IRC (Read error: Operation timed out) [05:14] *** Sk1d has quit IRC (Ping timeout: 250 seconds) [05:14] *** bwn has joined #archiveteam [05:14] *** metalcamp has joined #archiveteam [05:21] *** Sk1d has joined #archiveteam [05:22] *** metalcamp has quit IRC (Ping timeout: 258 seconds) [05:26] *** vitzli has quit IRC (Leaving) [05:26] *** vitzli has joined #archiveteam [05:29] *** VADemon has quit IRC (Quit: left4dead) [05:29] *** MMovie has quit IRC (Read error: Operation timed out) [05:31] *** MMovie has joined #archiveteam [05:37] re: pipeline direct to IA, could we just replace uploader.py with the script FOS uses to process archivebot warcs and replace the collection name? [05:38] i think that's what FalconK is working on [05:58] *** MMovie has quit IRC (Read error: Operation timed out) [05:59] *** MMovie has joined #archiveteam [06:17] *** MMovie has quit IRC (Read error: Operation timed out) [06:18] *** MMovie has joined #archiveteam [06:19] *** JesseW has quit IRC (Quit: Leaving.) [06:35] *** MMovie has quit IRC (Read error: Operation timed out) [06:35] *** MMovie has joined #archiveteam [06:53] *** Ungstein has joined #archiveteam [06:53] *** MMovie has quit IRC (Read error: Operation timed out) [06:54] *** MMovie has joined #archiveteam [06:55] *** Ungstein1 has quit IRC (Ping timeout: 260 seconds) [06:56] *** fie has quit IRC (Read error: Connection reset by peer) [07:11] *** MMovie has quit IRC (Read error: Operation timed out) [07:12] *** MMovie has joined #archiveteam [07:14] *** roninski has joined #archiveteam [07:14] *** ndizzle has joined #archiveteam [07:17] DOSBOX always works with sound on. It never is silent. [07:21] *** metalcamp has joined #archiveteam [07:25] *** MMovie has quit IRC (Read error: Operation timed out) [07:25] is there any chance someone's got a local copy of the fanfiction.net archive and can help me grab a particular file from it so i don't have to download the whole part? [07:25] Sure, which file are you looking for? [07:26] it's in part 9, lemme find the exact directory [07:27] *** xXx_ndidd has quit IRC (Read error: Operation timed out) [07:27] *** MMovie has joined #archiveteam [07:28] this folder: 1/18/183/u/1835454 [07:29] Hmm, you must be referring to a different fanfiction.net archive than the one I have [07:29] do you have the story archive one? [07:29] This is the one I have https://archive.org/details/FanfictionNearlyCompleteArchive [07:30] ahh yeah different one [07:31] i'm looking for this one: https://archive.org/details/archiveteam-fanfiction-warc-09 [07:31] thanks anyway :) [07:32] (unfortunately it's not a story i'm looking for it's a user profile) [07:36] *** RichardG has joined #archiveteam [07:41] roninski which file was it you were wanting to download [07:41] I could torrent it for you [07:44] 00000009.tar.megawarc.warc.gz from here - https://archive.org/download/archiveteam-fanfiction-warc-09, there's just a particular part of it i want (specifically the folder 1/18/183/u/1835454 within the archive) but i'm worried i'll kill my quota if i download the full file [07:46] unfortunately i'm not on unlimited but even though my quota is pretty big, considering how many people are in my household and how early in the month it is i'm not willing to risk it [07:46] considering the actual folder in the archive is probably tiny XD [07:50] quote on your interwebs? [07:50] roninski if you don't have it in ~5 hours highlight me and I'll grab it [07:51] 250gb on 250gb off 5 person household\ [07:51] thanks Fletcher :) [07:51] ahhhh sux bro, where do you live? I'm downloading the main file now, but will take a few hours :( [07:51] Australia [07:52] and thanks man, really appreciate it :) [07:53] *** WinterFox has joined #archiveteam [07:53] i'm moving to the US next month and should be able to get something better for my actual usage needs but until then I'm kinda stuck with terrible Aussie internet XD [07:53] where're you ErkDog? [07:55] US [07:56] where abouts? [07:57] Virginia [07:58] ah nice, i'm moving to Seattle [08:00] * xmc waves from seattle [08:01] *** MMovie has quit IRC (Read error: Operation timed out) [08:01] *** MMovie has joined #archiveteam [08:05] don't forget your umbrella!@!! [08:18] *** MMovie has quit IRC (Read error: Operation timed out) [08:20] *** MMovie has joined #archiveteam [08:22] *** atomotic has joined #archiveteam [08:24] *** dashcloud has quit IRC (Read error: Operation timed out) [08:28] *** dashcloud has joined #archiveteam [08:32] *** schbirid has joined #archiveteam [08:39] *** bwn has quit IRC (Read error: Operation timed out) [08:52] *** MMovie has quit IRC (Read error: Operation timed out) [08:53] *** MMovie has joined #archiveteam [08:57] *** metalcamp has quit IRC (Ping timeout: 258 seconds) [09:02] *** redlob has quit IRC (Quit: ZNC - http://znc.in) [09:03] *** redlob has joined #archiveteam [09:34] *** bwn has joined #archiveteam [09:37] *** MMovie has quit IRC (Read error: Operation timed out) [09:38] *** MMovie has joined #archiveteam [09:38] *** vtyl has quit IRC (Ping timeout: 250 seconds) [09:42] *** lytv has joined #archiveteam [10:09] *** MMovie has quit IRC (Read error: Operation timed out) [10:10] *** MMovie has joined #archiveteam [10:22] *** jut has joined #archiveteam [10:27] *** MMovie has quit IRC (Read error: Operation timed out) [10:29] *** MMovie has joined #archiveteam [10:47] *** MMovie has quit IRC (Read error: Operation timed out) [10:49] *** MMovie has joined #archiveteam [11:07] *** MMovie has quit IRC (Read error: Operation timed out) [11:07] *** MMovie has joined #archiveteam [11:09] *** metalcamp has joined #archiveteam [11:24] *** MMovie has quit IRC (Read error: Operation timed out) [11:26] *** MMovie has joined #archiveteam [11:43] *** MMovie has quit IRC (Read error: Operation timed out) [11:44] *** MMovie has joined #archiveteam [11:45] *** signius has quit IRC (Read error: Operation timed out) [11:49] *** signius has joined #archiveteam [11:50] *** dashcloud has quit IRC (Read error: Operation timed out) [11:54] *** dashcloud has joined #archiveteam [12:01] *** vOYtEC has quit IRC (Quit: rm -r *) [12:08] *** [phire] has quit IRC (Quit: ZNC - http://znc.in) [12:20] *** [phire] has joined #archiveteam [12:20] *** MMovie has quit IRC (Read error: Operation timed out) [12:22] *** MMovie has joined #archiveteam [12:27] *** Sk2d has joined #archiveteam [12:27] *** PurpleSym has quit IRC (*) [12:27] *** PurpleSym has joined #archiveteam [12:27] *** Sk1d has quit IRC (hub.se irc.du.se) [12:35] *** atomotic has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…) [12:42] *** MMovie has quit IRC (Read error: Operation timed out) [12:43] *** Sk2d is now known as Sk1d [12:43] *** metalcamp has quit IRC (Read error: Connection reset by peer) [12:44] *** MMovie has joined #archiveteam [12:45] *** metalcamp has joined #archiveteam [12:49] *** VADemon has joined #archiveteam [12:59] *** MMovie has quit IRC (Read error: Operation timed out) [12:59] *** WinterFox has quit IRC (Remote host closed the connection) [13:00] *** MMovie has joined #archiveteam [13:15] *** atomotic has joined #archiveteam [13:35] *** MMovie has quit IRC (Read error: Operation timed out) [13:35] *** MMovie has joined #archiveteam [13:38] *** dserodio has quit IRC (Quit: ZNC - http://znc.in) [13:52] *** MMovie has quit IRC (Read error: Operation timed out) [13:54] *** MMovie has joined #archiveteam [13:55] *** dserodio has joined #archiveteam [13:58] *** brayden_ has joined #archiveteam [13:58] *** swebb sets mode: +o brayden_ [14:02] *** brayden has quit IRC (Read error: Operation timed out) [14:12] *** pgoetz has quit IRC (Remote host closed the connection) [14:14] *** pgoetz has joined #archiveteam [14:14] *** pgoetz has quit IRC (Remote host closed the connection) [14:14] *** MMovie has quit IRC (Read error: Operation timed out) [14:15] *** MMovie has joined #archiveteam [14:32] *** MMovie has quit IRC (Read error: Operation timed out) [14:34] *** MMovie has joined #archiveteam [14:43] *** metalcamp has quit IRC (Ping timeout: 258 seconds) [14:44] The BBCshop.com is closing [14:44] http://www.bbcshop.com/page/helpfaq [15:00] *** MMovie has quit IRC (Read error: Operation timed out) [15:01] *** MMovie has joined #archiveteam [15:02] *** pgoetz has joined #archiveteam [15:08] *** metalcamp has joined #archiveteam [15:12] *** scyther has joined #archiveteam [15:35] *** dzman has joined #archiveteam [15:37] *** brayden_ is now known as brayden [15:37] *** MMovie has quit IRC (Read error: Operation timed out) [15:39] *** MMovie has joined #archiveteam [15:42] Guys, is there a(n easy) way finding a user in the hyves archive? :-) [15:44] not really [15:44] you can however [15:44] if you have the username find him or her on the url [15:45] How :)? [15:45] but there is not an easy way to search in the entire collection [15:45] add the url in the wayback machine [15:45] it will find it for you [15:46] What can i use in the url? [15:47] http://hyves.nl/username ? [15:47] the username if you know it [15:47] i think it was username.hyves.nl [15:47] ah thanks, i will try [15:48] good luck :) [15:49] *** MMovie has quit IRC (Read error: Operation timed out) [15:51] *** MMovie has joined #archiveteam [15:51] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [15:52] it says Page cannot be crawled or displayed due to robots.txt. every time i use my username :/ [16:00] well damn [16:00] *** MMovie has quit IRC (Read error: Operation timed out) [16:00] hyves.nl redirects to hyvesgames.nl which doesnt allow the archive.org crawler [16:00] thats shitty. [16:01] joepie91: did you notice this yet? [16:01] *** MMovie has joined #archiveteam [16:04] *** dzman has quit IRC (Ping timeout: 255 seconds) [16:18] I did not [16:20] I downloaded the CDX files for hyves. [16:20] And could grep them. [16:20] *** MMovie has quit IRC (Read error: Operation timed out) [16:21] *** MMovie has joined #archiveteam [16:24] *** metalcamp has quit IRC (Ping timeout: 258 seconds) [16:38] *** MMovie has quit IRC (Read error: Operation timed out) [16:40] *** MMovie has joined #archiveteam [16:51] *** bwn has quit IRC (Read error: Operation timed out) [16:54] *** atomotic has joined #archiveteam [16:56] *** MMovie has quit IRC (Read error: Operation timed out) [16:57] *** MMovie has joined #archiveteam [16:59] *** JesseW has joined #archiveteam [17:13] *** scyther has quit IRC (Quit: Leaving) [17:13] *** MMovie has quit IRC (Read error: Operation timed out) [17:14] *** MMovie has joined #archiveteam [17:22] *** JesseW has quit IRC (Quit: Leaving.) [17:31] *** MMovie has quit IRC (Read error: Operation timed out) [17:32] *** MMovie has joined #archiveteam [17:33] *** xXx_ndidd has joined #archiveteam [17:38] *** bwn has joined #archiveteam [17:46] *** ndizzle has quit IRC (Read error: Operation timed out) [17:47] *** MMovie has quit IRC (Read error: Operation timed out) [17:48] *** MMovie has joined #archiveteam [18:06] *** MMovie has quit IRC (Read error: Operation timed out) [18:07] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [18:08] *** MMovie has joined #archiveteam [18:13] *** vitzli has quit IRC (Leaving) [18:16] *** ndizzle has joined #archiveteam [18:29] *** xXx_ndidd has quit IRC (Read error: Operation timed out) [18:35] *** MMovie has quit IRC (Read error: Operation timed out) [18:36] *** MMovie has joined #archiveteam [18:45] *** HCross has quit IRC (Read error: Connection reset by peer) [18:49] *** HCross has joined #archiveteam [19:05] *** yipdw has quit IRC (Ping timeout: 1224 seconds) [19:05] *** signius has quit IRC (Ping timeout: 345 seconds) [19:05] *** FalconK has quit IRC (Ping timeout: 345 seconds) [19:05] *** hive-mind has quit IRC (Ping timeout: 316 seconds) [19:05] *** SirCmpwn has quit IRC (Ping timeout: 345 seconds) [19:05] *** bauruine has quit IRC (Ping timeout: 316 seconds) [19:05] *** ploopkaz- has quit IRC (Ping timeout: 345 seconds) [19:05] *** Atluxity has quit IRC (Ping timeout: 345 seconds) [19:05] *** balrog has quit IRC (Ping timeout: 345 seconds) [19:05] *** dan- has quit IRC (Ping timeout: 345 seconds) [19:05] *** HCross2 has quit IRC (Read error: Connection reset by peer) [19:05] *** johtso has quit IRC (Read error: Connection reset by peer) [19:05] *** zhongfu has quit IRC (Remote host closed the connection) [19:05] *** wp494_ has joined #archiveteam [19:05] *** victor has quit IRC (Write error: Broken pipe) [19:05] *** d_rebel has quit IRC (Write error: Connection reset by peer) [19:05] *** Vito` has quit IRC (Write error: Connection reset by peer) [19:05] *** winr4r has quit IRC (Write error: Connection reset by peer) [19:05] *** victor has joined #archiveteam [19:05] *** Vito` has joined #archiveteam [19:05] *** _desu____ has joined #archiveteam [19:05] *** hive-mind has joined #archiveteam [19:05] *** bauruine has joined #archiveteam [19:06] *** d_rebel has joined #archiveteam [19:06] *** Boltsie_ has joined #archiveteam [19:06] *** balrog has joined #archiveteam [19:06] *** swebb sets mode: +o balrog [19:06] *** ploopkazo has joined #archiveteam [19:06] *** deathy_ has joined #archiveteam [19:06] *** Atluxity has joined #archiveteam [19:06] *** TheKiwi_ has joined #archiveteam [19:06] *** Ungstein1 has joined #archiveteam [19:06] *** FalconK has joined #archiveteam [19:06] *** beeper_ has joined #archiveteam [19:07] *** casdr_ has joined #archiveteam [19:07] *** casdr_ has quit IRC (Connection closed) [19:07] *** beeper_ has quit IRC (Connection closed) [19:07] *** TheKiwi_ has quit IRC (Connection closed) [19:07] *** kevin_ has joined #archiveteam [19:07] *** casdr_ has joined #archiveteam [19:07] *** TheKiwi_ has joined #archiveteam [19:07] *** beeper_ has joined #archiveteam [19:07] *** zhongfu has joined #archiveteam [19:08] *** Ungstein has quit IRC (Ping timeout: 274 seconds) [19:08] *** wp494 has quit IRC (Ping timeout: 274 seconds) [19:08] *** _desu___ has quit IRC (Ping timeout: 274 seconds) [19:08] *** Boltsie has quit IRC (Ping timeout: 274 seconds) [19:08] *** JSharp___ has quit IRC (Ping timeout: 274 seconds) [19:08] *** TheKiwi has quit IRC (Ping timeout: 274 seconds) [19:08] *** Ctrl-S___ has quit IRC (Ping timeout: 274 seconds) [19:08] *** beeper has quit IRC (Ping timeout: 274 seconds) [19:08] *** kevin has quit IRC (Ping timeout: 274 seconds) [19:08] *** casdr has quit IRC (Ping timeout: 274 seconds) [19:08] *** abartov__ has quit IRC (Ping timeout: 274 seconds) [19:08] *** karissa__ has quit IRC (Ping timeout: 274 seconds) [19:08] *** VonGuard has quit IRC (Ping timeout: 274 seconds) [19:08] *** deathy has quit IRC (Ping timeout: 274 seconds) [19:08] *** _desu____ is now known as _desu___ [19:08] *** Boltsie_ is now known as Boltsie [19:08] *** deathy_ is now known as deathy [19:08] *** SirCmpwn has joined #archiveteam [19:08] *** casdr_ is now known as casdr [19:08] *** beeper_ is now known as beeper [19:08] *** TheKiwi_ is now known as TheKiwi [19:08] *** kevin_ is now known as kevin [19:08] *** abartov__ has joined #archiveteam [19:08] *** dan- has joined #archiveteam [19:08] *** HCross2 has joined #archiveteam [19:09] *** beeper has quit IRC (Remote host closed the connection) [19:09] *** beeper has joined #archiveteam [19:10] *** johtso has joined #archiveteam [19:10] *** signius has joined #archiveteam [19:10] *** TheKiwi has quit IRC (Remote host closed the connection) [19:11] *** TheKiwi has joined #archiveteam [19:12] *** winr4r has joined #archiveteam [19:23] *** jut has quit IRC (jut) [19:30] *** Tomcat_ has joined #archiveteam [19:33] *** Tomcat__ has joined #archiveteam [19:35] *** Tomcat_ has quit IRC (Read error: Operation timed out) [19:56] Fletcher: code should drop this week [20:05] *** MMovie has quit IRC (Read error: Operation timed out) [20:05] *** Tomcat__ has quit IRC (Remote host closed the connection) [20:06] *** MMovie has joined #archiveteam [20:22] *** MMovie has quit IRC (Read error: Operation timed out) [20:24] *** MMovie has joined #archiveteam [20:29] *** ndiddy has joined #archiveteam [20:30] *** ndizzle has quit IRC (Read error: Operation timed out) [20:39] *** dashcloud has quit IRC (Read error: Operation timed out) [20:42] *** dashcloud has joined #archiveteam [20:49] *** MMovie has quit IRC (Read error: Operation timed out) [20:51] *** MMovie has joined #archiveteam [21:01] *** metalcamp has joined #archiveteam [21:06] *** MMovie has quit IRC (Read error: Operation timed out) [21:08] *** MMovie has joined #archiveteam [21:20] *** jake1 has joined #archiveteam [21:21] *** MMovie has quit IRC (Read error: Operation timed out) [21:23] *** MMovie has joined #archiveteam [21:24] MrRadar: SketchCow passed along the OverflowError bug you found in the ia CLI when uploading files larger than ~2GB. This should fix it here: https://github.com/jjjake/internetarchive/commit/6c9f77cb9b57296bc88278b5716c7a3bc32c3b43 [21:25] That fix will be in v1.0.2, which I hope to release later today. [21:33] *** schbirid has quit IRC (Quit: Leaving) [21:34] * JW_work waves to jake1 :-) [21:46] *** MMovie has quit IRC (Read error: Operation timed out) [21:48] *** MMovie has joined #archiveteam [21:56] *** fie has joined #archiveteam [22:01] Thanks jake1! [22:07] *** dashcloud has quit IRC (Read error: Operation timed out) [22:08] jake1: Thanks for the fix. [22:09] *** metalcamp has quit IRC (Ping timeout: 258 seconds) [22:11] *** dashcloud has joined #archiveteam [22:24] *** MMovie has quit IRC (Read error: Operation timed out) [22:25] *** MMovie has joined #archiveteam [22:42] *** MMovie has quit IRC (Read error: Operation timed out) [22:43] *** MMovie has joined #archiveteam [22:47] OK so I told roninski I would help him get a folder out of a web archive, so I downloaded it and extracted it, so now I have this 138 Gig warc file that I don't know what to do with :( [22:49] You can use warcat to extract it [22:49] https://pypi.python.org/pypi/Warcat/ [22:49] There are other tools as well [22:49] that's pythony, I'm in windows :( [22:51] *** yipdw has joined #archiveteam [22:52] this looks promising: https://github.com/ikreymer/webarchiveplayer [22:52] Well, that will let you browse the web archive (like the IA's Wayback Machine) [22:52] You can install Python on Windows [22:54] *** wp494_ is now known as wp494 [22:55] ohhhh yeah I guess I can [22:56] lol it would be cool if I could run the scripts from windows command line instead of loosing all these resources to the virtual box hypervisor, lol [22:57] ErkDog: and actually, there's an instance of (a variant of) webarchiveplayer running at http://archivelab.org:3579/item/{IA identifier} so if you put the archive identifier you downloaded in at the end, then /*/ then the website, you should be able to get it without even downloading it. [22:59] *** JW_work has left [22:59] *** JW_work has joined #archiveteam [22:59] OOOO'rly thanks JW_work [22:59] *** MMovie has quit IRC (Read error: Operation timed out) [22:59] so, webarchiveplayer is also bundled as a Windows application [23:01] *** MMovie has joined #archiveteam [23:01] what constitutes the "IA Identifier" of : https://archive.org/download/archiveteam-fanfiction-warc-09 [23:01] *** RedType has left [23:02] archiveteam-fanfiction-warc-09 [23:17] *** VADemon has quit IRC (Quit: left4dead) [23:19] *** MMovie has quit IRC (Read error: Operation timed out) [23:21] *** MMovie has joined #archiveteam [23:38] *** MMovie has quit IRC (Read error: Operation timed out) [23:40] *** MMovie has joined #archiveteam [23:56] *** MMovie has quit IRC (Read error: Operation timed out) [23:57] *** MMovie has joined #archiveteam