#archiveteam 2016-03-07,Mon

↑back Search

Time Nickname Message
00:06 🔗 MMovie has quit IRC (Read error: Operation timed out)
00:07 🔗 MMovie has joined #archiveteam
00:22 🔗 Stilett0 is now known as Stiletto
00:35 🔗 redlob has quit IRC (Read error: Operation timed out)
00:37 🔗 dashcloud has quit IRC (Read error: Operation timed out)
00:39 🔗 dashcloud has joined #archiveteam
00:45 🔗 redlob has joined #archiveteam
00:53 🔗 MMovie has quit IRC (Read error: Operation timed out)
00:55 🔗 MMovie has joined #archiveteam
01:08 🔗 MMovie has quit IRC (Read error: Operation timed out)
01:09 🔗 MMovie has joined #archiveteam
01:11 🔗 JesseW has joined #archiveteam
01:12 🔗 FalconK so FOS is still looking really congested
01:12 🔗 FalconK like 50-100kB/s even after connections were limited to 50 concurrent
01:12 🔗 FalconK is there some way I can configure it to assemble ~25GB megawarcs and upload more directly to IA?
01:22 🔗 yipdw yeah, you'll want the archiveteam-megawarc-factory git repo, an IA account, IA-S3 keys (https://archive.org/account/s3.php) and ideally access to the collection you want to put stuff in
01:22 🔗 yipdw I can assist with setup of the whole smash except for the last bit, though I am booked until tomorrow
01:23 🔗 yipdw I think there must be others who know how it works, though -- xmc chfoo ersi joepie91 maybe
01:23 🔗 yipdw or they can be dereferenced
01:23 🔗 xmc yes hi
01:23 🔗 FalconK yeah xmc is to my left.
01:23 🔗 n00b599 has joined #archiveteam
01:23 🔗 dashcloud has quit IRC (Read error: Operation timed out)
01:23 🔗 yipdw wait, physically
01:23 🔗 n00b599 r/ring secret word.
01:23 🔗 * xmc waves
01:23 🔗 FalconK I can do the software thing but the account and keys and access I obviously need help with :)
01:24 🔗 FalconK not super urgent - tomorrow is just fine
01:24 🔗 dashcloud has joined #archiveteam
01:24 🔗 yipdw IA account is just any IA account, keys can be generated at that link
01:24 🔗 yipdw the access thing, ping SketchCow
01:24 🔗 * FalconK pings at SketchCow
01:24 🔗 n00b599 Didn't work.
01:25 🔗 n00b599 Oh got it
01:25 🔗 xmc yahoosucks
01:26 🔗 n00b599 Thank ya
01:27 🔗 xmc sure thing
01:27 🔗 xmc what're you up to?
01:29 🔗 Anon has joined #archiveteam
01:29 🔗 Anon has quit IRC (Client Quit)
01:30 🔗 Anon has joined #archiveteam
01:31 🔗 Anon loveisover is down.
01:31 🔗 Anon has quit IRC (Client Quit)
01:40 🔗 philpem has quit IRC (Ping timeout: 260 seconds)
01:47 🔗 FalconK can I not just upload things to the archivebot collection?
01:47 🔗 FalconK perhaps I will try that
01:49 🔗 xmc i think that perhaps we should consider giving each pipeline operator a sub-collection inside of archivebot, or maybe just privileges to upload into it
01:49 🔗 xmc hm
01:49 🔗 xmc i'm not sure
01:49 🔗 xmc there are many ways that we could do this differently
01:53 🔗 yipdw oh
01:53 🔗 yipdw archivebot collection
01:53 🔗 yipdw yeah I think SketchCow's the only one who has that
01:53 🔗 * yipdw doesn't have that access
01:54 🔗 xmc yeah
01:54 🔗 xmc sounds about right
01:54 🔗 xmc we've been talking about this for a little while and i still can't think of a good reason for archivebot to depend on fos
02:01 🔗 n00b599 Oh were you talking to me at 19:27?
02:02 🔗 xmc it's not even 19:00, friend
02:03 🔗 MrRadar Central time zone best time zone
02:03 🔗 * MrRadar puts on shades
02:04 🔗 n00b599 YEEEEEEEEEEEEEEEEEEEAAAAAAAAAAAAAAAAAH!!!
02:05 🔗 n00b599 I'm from Miami for real, so I guess you can say that meme comes with the nativety.
02:06 🔗 n00b599 It's arguably as good or better than Ocarina of Time in some ways. In some ways it's not. I beat it 10 years ago on GCN and don't remember any of it so I've been playing it on the 3DS and it's awesome. I've died and reset the game from the beginning twice in the Water Temple already (the first time I ran out of oxygen in a room where I couldn't figure out what to do and the second time I got killed by Stalfos) so I have
02:06 🔗 xmc woop woop woop off-topic siren
02:06 🔗 ErkDog Arguabely, it's better to have a single collection point, so you aren't dealing with maintaining multiple systems
02:06 🔗 n00b599 I was testing how the IRC formatted.. That's what I was typing out when you asked what I was up to.
02:07 🔗 xmc ErkDog: well it's overloaded and slowing down archivebot
02:07 🔗 n00b599 Do you guys 4chan?
02:08 🔗 xmc why do you ask?
02:08 🔗 n00b599 That's what I signed up here for. That and I'm an archivist in my own right. I'm surprised I never heard of this site.
02:08 🔗 xmc are you ANOTHER 4chan archiver?
02:08 🔗 n00b599 I
02:08 🔗 n00b599 I
02:08 🔗 n00b599 I'm looking for the data from May to October 4th.
02:09 🔗 n00b599 Negatory.
02:09 🔗 xmc for what
02:09 🔗 n00b599 Just a frequent poster who had a lot of posts I wanted to reread.
02:09 🔗 xmc beats me. we don't archive 4chan actively.
02:09 🔗 n00b599 Hisotircal preservation on the note as well.
02:10 🔗 n00b599 Ah, I see. What I'm referring to is a good Samaritan that I read about on the 4chanarchive page who had a private archive and was volunteering to share it.
02:10 🔗 xmc no, this is not an archive warez channel, we don't keep a meat-index of stuff
02:10 🔗 xmc all I can say is http://archive.org/search.php
02:11 🔗 n00b599 Oh that's what you're affiliated directly with?
02:11 🔗 xmc no
02:11 🔗 MrRadar We use the IA as a repository for our work but there is no official connection
02:11 🔗 xmc but things wind up there
02:11 🔗 n00b599 Gotcha
02:12 🔗 MrRadar If you know someone with archived data to share, ask him to upload it there
02:12 🔗 MrRadar That way it will be preserved
02:12 🔗 n00b599 Well. if nobody's heard what I'm talking about, I think I found what I need. I'll just make a section on the discussion page/
02:12 🔗 n00b599 *nod*nod*
02:24 🔗 lokis has joined #archiveteam
02:24 🔗 n00b599 o/
02:25 🔗 MMovie has quit IRC (Read error: Operation timed out)
02:26 🔗 MMovie has joined #archiveteam
02:27 🔗 ErkDog yeah it is XMC, lol
02:28 🔗 ErkDog I mentioned previously that it might be a good idea to have reginal RSYnc systems so like for instance, I could pick a target in the US
02:28 🔗 ErkDog EU people could pick EU
02:29 🔗 ErkDog then those targets could single threaded push stuff over to FOS so it wasn't gettings it's IO thrashed by 50 simultaneous RSYNCHs coming in
02:29 🔗 ErkDog but instead 2 or 3
02:29 🔗 ErkDog and then the back end could keep up
02:29 🔗 FalconK there is no point in using rsync for this at all
02:29 🔗 FalconK since it looks like FOS is merely forwarding the archives as-is into IA collection archivebot
02:29 🔗 FalconK just send them directly using the HTTP REST API
02:30 🔗 xmc i don't think ia supports giving multiple users access to a restricted collection?
02:30 🔗 FalconK like bouncing them through FOS is a waste of bandwidth
02:30 🔗 ErkDog not really because archive bot is on the same network
02:30 🔗 xmc but subcollections, one per archivebot pipeline owner, make sense
02:30 🔗 ErkDog have you ever tracerouted FOS?
02:30 🔗 FalconK yes, it's in IA
02:30 🔗 FalconK that is fine
02:30 🔗 FalconK it's clearly overloaded, and a bottleneck, and it doesn't need to be so it shou;dn't be
02:31 🔗 FalconK we're not saturating IA downstream
02:31 🔗 FalconK or our upstream
02:31 🔗 ErkDog yes, but that's because FOS is dealing with 50 incoming rsynch's
02:31 🔗 xmc point of order, i would say "inbound" and "outbound"
02:31 🔗 ErkDog instead of 2 or 3 from front end systems
02:31 🔗 FalconK so it doesn't matter why
02:31 🔗 xmc the bottleneck is in FOS's disk bandwidth
02:31 🔗 FalconK there is just literally no reason to have that extra infrastructure
02:32 🔗 ErkDog well if Archive Bot could pull the data from the other RSynch servers, then sure
02:32 🔗 FalconK why use rsync at all?
02:32 🔗 ErkDog because rsync does CRC checking
02:32 🔗 ErkDog HTTP and standard FTP don't
02:33 🔗 FalconK the BER is probably the same, since FOS uploads over a network too
02:33 🔗 FalconK also doesn't WARC have a CRC?
02:34 🔗 ErkDog I wouldn't call a transfer inside the same building a "network" upload that's succeptible to the same potential for corruption as sending data across the open internet
02:34 🔗 ErkDog I'm not sure if WARC does or not, but if it did, you'd have to do the CRC after transmit, and then retransmit the entire WARC file if it failed, rsync does the CRC in real time as things are transmitted
02:34 🔗 ErkDog why do you h8 rsync? did it murder your family?
02:36 🔗 xmc TCP has a CRC
02:36 🔗 xmc on every packet!
02:36 🔗 MrRadar And yes WARC files have checksums on each record too
02:36 🔗 xmc it's part of the error control mechanism of the internet at large
02:36 🔗 xmc it works pretty well
02:36 🔗 n00b599 I got a bad vibe.
02:36 🔗 xmc i've never received an accidentally corrupted tcp stream
02:37 🔗 ErkDog ohhh good point TCP does
02:37 🔗 n00b599 What;s a TCP screen?
02:37 🔗 yipdw so
02:37 🔗 yipdw if someone wants to write a different uploader for ArchiveBot, I am down with that
02:37 🔗 ErkDog however, that doesn't change the fact, that I've personally seen, and experienced FTP and HTTP transfers, using TCP which ended up corrupted and required re-transmittal
02:37 🔗 FalconK I am going to, yipdw :)
02:37 🔗 FalconK already in progress :)
02:37 🔗 yipdw ideally, it would just require replacement of uploader.py
02:38 🔗 FalconK yup
02:38 🔗 xmc there. blessings from two people with @ and i'm sure SketchCow will be happy to get it off FOS
02:38 🔗 marvinw_ is now known as ivan`
02:38 🔗 yipdw it would however require each operator to have upload privileges into the archivebot collection, as well as a way to generate item names
02:38 🔗 xmc is why i propose a subcollection for operators
02:39 🔗 xmc though that would mean that whoever is in charge of archivebot have permissions to create these collections
02:39 🔗 xmc my knowledge of IA's permissions mechanisms is, sadly, lacking
02:39 🔗 yipdw anyway we'll get the tool first and figure out access later
02:39 🔗 * xmc nods
02:39 🔗 yipdw and are people seriously talking about TCP checksums and corrupted data w tf
02:40 🔗 * yipdw alt+tab
02:40 🔗 ErkDog lol yipdw
02:40 🔗 xmc ALSO
02:40 🔗 xmc having the pipeline operator upload directly actually gives us a much better audit trail
02:40 🔗 xmc which is a thing i've been thinking about for a little while
02:41 🔗 ErkDog Well without changing the existing access permissions, having regional front end rsynch targets which then sent single threaded to FOS would lessen it's disk I/O significantly.
02:41 🔗 ErkDog 50 RSynchs run more than 50 times slower than 1, even 2 or 3
02:41 🔗 xmc why are you so stuck on rsync
02:41 🔗 ErkDog well cause that's just how it works now
02:41 🔗 yipdw so
02:41 🔗 yipdw we've seen this behavior, yes
02:41 🔗 ErkDog and would be easier to change the work flow
02:41 🔗 xmc it works that way now because when we wrote it we were feeling lazy
02:42 🔗 ErkDog than having IA give access permissions that they may or may not be willing to do
02:42 🔗 xmc sometimes it is the correct time to redesign things
02:42 🔗 xmc now is, apparently, the correct time
02:42 🔗 MrRadar SketchCow can sort out any IA permissions we need. It's not an issue
02:42 🔗 yipdw well that and if you give each operator subcollections the problem goes away, ish
02:42 🔗 ErkDog OMG, if I could submit work loads at faster than 100K/sec I would be immensely happy :-D
02:42 🔗 FalconK yes
02:43 🔗 FalconK we share a common goal :D
02:43 🔗 yipdw this isn't an invitation to shove fucking en.wikipedia.org into archivebot
02:43 🔗 FalconK !a en.wikipedia.org
02:43 🔗 FalconK er
02:43 🔗 FalconK :P
02:43 🔗 xmc you need http://
02:43 🔗 FalconK I think you can just download their whole SQL database anyway. no reason to crawl it.
02:43 🔗 xmc that's what WIKITEAM is for
02:43 🔗 xmc i think Nemo_bis is in charge of that
02:44 🔗 yipdw come to think of it, ia upload plus shell script is about all you need
02:44 🔗 yipdw well
02:44 🔗 yipdw and a 64-bit system
02:44 🔗 FalconK ha
02:45 🔗 xmc all that and a bag of chips
02:45 🔗 yipdw I mean curl or whatever, it doesn't matter really
02:52 🔗 dashcloud has quit IRC (Read error: Operation timed out)
02:53 🔗 FalconK I'
02:54 🔗 mismatch_ has quit IRC (Remote host closed the connection)
02:54 🔗 FalconK once we get permissions set up I can test my thing
02:54 🔗 mismatch_ has joined #archiveteam
02:56 🔗 dashcloud has joined #archiveteam
03:09 🔗 MMovie has quit IRC (Read error: Operation timed out)
03:10 🔗 MMovie has joined #archiveteam
03:11 🔗 n00b599 has quit IRC (Quit: Page closed)
03:22 🔗 superkuh has quit IRC (Quit: the neuronal action potential is an electrical manipulation of reversible abrupt phase changes in the lipid bilaye)
03:23 🔗 MMovie has quit IRC (Read error: Operation timed out)
03:24 🔗 MMovie has joined #archiveteam
03:37 🔗 bwn has quit IRC (Read error: Operation timed out)
03:37 🔗 MMovie has quit IRC (Read error: Operation timed out)
03:38 🔗 MMovie has joined #archiveteam
03:40 🔗 Stiletto has quit IRC (Read error: Connection reset by peer)
03:41 🔗 Stolett0 has joined #archiveteam
03:44 🔗 JesseW SketchCow: BTW, when I try to turn on sound on https://archive.org/details/msdos_Alpine_Tram_Ride_1989 it tells me: "This button only works once the emulation is running" even after the emulator appears to be running. IDK if I should email this to you, to info, both or neither.
03:44 🔗 Stolett0 is now known as Stiletto
03:48 🔗 Stolett0 has joined #archiveteam
03:58 🔗 Stiletto has quit IRC (Read error: Operation timed out)
03:59 🔗 vitzli has joined #archiveteam
03:59 🔗 MMovie has quit IRC (Read error: Operation timed out)
04:00 🔗 Stolett0 has quit IRC (Read error: Connection reset by peer)
04:00 🔗 MMovie has joined #archiveteam
04:01 🔗 Stolett0 has joined #archiveteam
04:17 🔗 MMovie has quit IRC (Read error: Operation timed out)
04:19 🔗 MMovie has joined #archiveteam
04:25 🔗 Stolett0 is now known as Stiletto
04:26 🔗 Stiletto is now known as Stilett0
04:26 🔗 Stilett0 is now known as Stiletto
04:29 🔗 tomwsmf-a has quit IRC (Read error: Operation timed out)
04:44 🔗 MMovie has quit IRC (Read error: Operation timed out)
04:45 🔗 MMovie has joined #archiveteam
05:02 🔗 MMovie has quit IRC (Read error: Operation timed out)
05:04 🔗 MMovie has joined #archiveteam
05:07 🔗 RedType has joined #archiveteam
05:07 🔗 RichardG has quit IRC (Read error: Operation timed out)
05:14 🔗 Sk1d has quit IRC (Ping timeout: 250 seconds)
05:14 🔗 bwn has joined #archiveteam
05:14 🔗 metalcamp has joined #archiveteam
05:21 🔗 Sk1d has joined #archiveteam
05:22 🔗 metalcamp has quit IRC (Ping timeout: 258 seconds)
05:26 🔗 vitzli has quit IRC (Leaving)
05:26 🔗 vitzli has joined #archiveteam
05:29 🔗 VADemon has quit IRC (Quit: left4dead)
05:29 🔗 MMovie has quit IRC (Read error: Operation timed out)
05:31 🔗 MMovie has joined #archiveteam
05:37 🔗 Fletcher re: pipeline direct to IA, could we just replace uploader.py with the script FOS uses to process archivebot warcs and replace the collection name?
05:38 🔗 xmc i think that's what FalconK is working on
05:58 🔗 MMovie has quit IRC (Read error: Operation timed out)
05:59 🔗 MMovie has joined #archiveteam
06:17 🔗 MMovie has quit IRC (Read error: Operation timed out)
06:18 🔗 MMovie has joined #archiveteam
06:19 🔗 JesseW has quit IRC (Quit: Leaving.)
06:35 🔗 MMovie has quit IRC (Read error: Operation timed out)
06:35 🔗 MMovie has joined #archiveteam
06:53 🔗 Ungstein has joined #archiveteam
06:53 🔗 MMovie has quit IRC (Read error: Operation timed out)
06:54 🔗 MMovie has joined #archiveteam
06:55 🔗 Ungstein1 has quit IRC (Ping timeout: 260 seconds)
06:56 🔗 fie has quit IRC (Read error: Connection reset by peer)
07:11 🔗 MMovie has quit IRC (Read error: Operation timed out)
07:12 🔗 MMovie has joined #archiveteam
07:14 🔗 roninski has joined #archiveteam
07:14 🔗 ndizzle has joined #archiveteam
07:17 🔗 SketchCow DOSBOX always works with sound on. It never is silent.
07:21 🔗 metalcamp has joined #archiveteam
07:25 🔗 MMovie has quit IRC (Read error: Operation timed out)
07:25 🔗 roninski is there any chance someone's got a local copy of the fanfiction.net archive and can help me grab a particular file from it so i don't have to download the whole part?
07:25 🔗 MrRadar Sure, which file are you looking for?
07:26 🔗 roninski it's in part 9, lemme find the exact directory
07:27 🔗 xXx_ndidd has quit IRC (Read error: Operation timed out)
07:27 🔗 MMovie has joined #archiveteam
07:28 🔗 roninski this folder: 1/18/183/u/1835454
07:29 🔗 MrRadar Hmm, you must be referring to a different fanfiction.net archive than the one I have
07:29 🔗 roninski do you have the story archive one?
07:29 🔗 MrRadar This is the one I have https://archive.org/details/FanfictionNearlyCompleteArchive
07:30 🔗 roninski ahh yeah different one
07:31 🔗 roninski i'm looking for this one: https://archive.org/details/archiveteam-fanfiction-warc-09
07:31 🔗 roninski thanks anyway :)
07:32 🔗 roninski (unfortunately it's not a story i'm looking for it's a user profile)
07:36 🔗 RichardG has joined #archiveteam
07:41 🔗 ErkDog roninski which file was it you were wanting to download
07:41 🔗 ErkDog I could torrent it for you
07:44 🔗 roninski 00000009.tar.megawarc.warc.gz from here - https://archive.org/download/archiveteam-fanfiction-warc-09, there's just a particular part of it i want (specifically the folder 1/18/183/u/1835454 within the archive) but i'm worried i'll kill my quota if i download the full file
07:46 🔗 roninski unfortunately i'm not on unlimited but even though my quota is pretty big, considering how many people are in my household and how early in the month it is i'm not willing to risk it
07:46 🔗 roninski considering the actual folder in the archive is probably tiny XD
07:50 🔗 ErkDog quote on your interwebs?
07:50 🔗 Fletcher roninski if you don't have it in ~5 hours highlight me and I'll grab it
07:51 🔗 roninski 250gb on 250gb off 5 person household\
07:51 🔗 roninski thanks Fletcher :)
07:51 🔗 ErkDog ahhhh sux bro, where do you live? I'm downloading the main file now, but will take a few hours :(
07:51 🔗 roninski Australia
07:52 🔗 roninski and thanks man, really appreciate it :)
07:53 🔗 WinterFox has joined #archiveteam
07:53 🔗 roninski i'm moving to the US next month and should be able to get something better for my actual usage needs but until then I'm kinda stuck with terrible Aussie internet XD
07:53 🔗 roninski where're you ErkDog?
07:55 🔗 ErkDog US
07:56 🔗 roninski where abouts?
07:57 🔗 ErkDog Virginia
07:58 🔗 roninski ah nice, i'm moving to Seattle
08:00 🔗 * xmc waves from seattle
08:01 🔗 MMovie has quit IRC (Read error: Operation timed out)
08:01 🔗 MMovie has joined #archiveteam
08:05 🔗 ErkDog don't forget your umbrella!@!!
08:18 🔗 MMovie has quit IRC (Read error: Operation timed out)
08:20 🔗 MMovie has joined #archiveteam
08:22 🔗 atomotic has joined #archiveteam
08:24 🔗 dashcloud has quit IRC (Read error: Operation timed out)
08:28 🔗 dashcloud has joined #archiveteam
08:32 🔗 schbirid has joined #archiveteam
08:39 🔗 bwn has quit IRC (Read error: Operation timed out)
08:52 🔗 MMovie has quit IRC (Read error: Operation timed out)
08:53 🔗 MMovie has joined #archiveteam
08:57 🔗 metalcamp has quit IRC (Ping timeout: 258 seconds)
09:02 🔗 redlob has quit IRC (Quit: ZNC - http://znc.in)
09:03 🔗 redlob has joined #archiveteam
09:34 🔗 bwn has joined #archiveteam
09:37 🔗 MMovie has quit IRC (Read error: Operation timed out)
09:38 🔗 MMovie has joined #archiveteam
09:38 🔗 vtyl has quit IRC (Ping timeout: 250 seconds)
09:42 🔗 lytv has joined #archiveteam
10:09 🔗 MMovie has quit IRC (Read error: Operation timed out)
10:10 🔗 MMovie has joined #archiveteam
10:22 🔗 jut has joined #archiveteam
10:27 🔗 MMovie has quit IRC (Read error: Operation timed out)
10:29 🔗 MMovie has joined #archiveteam
10:47 🔗 MMovie has quit IRC (Read error: Operation timed out)
10:49 🔗 MMovie has joined #archiveteam
11:07 🔗 MMovie has quit IRC (Read error: Operation timed out)
11:07 🔗 MMovie has joined #archiveteam
11:09 🔗 metalcamp has joined #archiveteam
11:24 🔗 MMovie has quit IRC (Read error: Operation timed out)
11:26 🔗 MMovie has joined #archiveteam
11:43 🔗 MMovie has quit IRC (Read error: Operation timed out)
11:44 🔗 MMovie has joined #archiveteam
11:45 🔗 signius has quit IRC (Read error: Operation timed out)
11:49 🔗 signius has joined #archiveteam
11:50 🔗 dashcloud has quit IRC (Read error: Operation timed out)
11:54 🔗 dashcloud has joined #archiveteam
12:01 🔗 vOYtEC has quit IRC (Quit: rm -r *)
12:08 🔗 [phire] has quit IRC (Quit: ZNC - http://znc.in)
12:20 🔗 [phire] has joined #archiveteam
12:20 🔗 MMovie has quit IRC (Read error: Operation timed out)
12:22 🔗 MMovie has joined #archiveteam
12:27 🔗 Sk2d has joined #archiveteam
12:27 🔗 PurpleSym has quit IRC (*)
12:27 🔗 PurpleSym has joined #archiveteam
12:27 🔗 Sk1d has quit IRC (hub.se irc.du.se)
12:35 🔗 atomotic has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…)
12:42 🔗 MMovie has quit IRC (Read error: Operation timed out)
12:43 🔗 Sk2d is now known as Sk1d
12:43 🔗 metalcamp has quit IRC (Read error: Connection reset by peer)
12:44 🔗 MMovie has joined #archiveteam
12:45 🔗 metalcamp has joined #archiveteam
12:49 🔗 VADemon has joined #archiveteam
12:59 🔗 MMovie has quit IRC (Read error: Operation timed out)
12:59 🔗 WinterFox has quit IRC (Remote host closed the connection)
13:00 🔗 MMovie has joined #archiveteam
13:15 🔗 atomotic has joined #archiveteam
13:35 🔗 MMovie has quit IRC (Read error: Operation timed out)
13:35 🔗 MMovie has joined #archiveteam
13:38 🔗 dserodio has quit IRC (Quit: ZNC - http://znc.in)
13:52 🔗 MMovie has quit IRC (Read error: Operation timed out)
13:54 🔗 MMovie has joined #archiveteam
13:55 🔗 dserodio has joined #archiveteam
13:58 🔗 brayden_ has joined #archiveteam
13:58 🔗 swebb sets mode: +o brayden_
14:02 🔗 brayden has quit IRC (Read error: Operation timed out)
14:12 🔗 pgoetz has quit IRC (Remote host closed the connection)
14:14 🔗 pgoetz has joined #archiveteam
14:14 🔗 pgoetz has quit IRC (Remote host closed the connection)
14:14 🔗 MMovie has quit IRC (Read error: Operation timed out)
14:15 🔗 MMovie has joined #archiveteam
14:32 🔗 MMovie has quit IRC (Read error: Operation timed out)
14:34 🔗 MMovie has joined #archiveteam
14:43 🔗 metalcamp has quit IRC (Ping timeout: 258 seconds)
14:44 🔗 HCross The BBCshop.com is closing
14:44 🔗 HCross http://www.bbcshop.com/page/helpfaq
15:00 🔗 MMovie has quit IRC (Read error: Operation timed out)
15:01 🔗 MMovie has joined #archiveteam
15:02 🔗 pgoetz has joined #archiveteam
15:08 🔗 metalcamp has joined #archiveteam
15:12 🔗 scyther has joined #archiveteam
15:35 🔗 dzman has joined #archiveteam
15:37 🔗 brayden_ is now known as brayden
15:37 🔗 MMovie has quit IRC (Read error: Operation timed out)
15:39 🔗 MMovie has joined #archiveteam
15:42 🔗 dzman Guys, is there a(n easy) way finding a user in the hyves archive? :-)
15:44 🔗 midas not really
15:44 🔗 midas you can however
15:44 🔗 midas if you have the username find him or her on the url
15:45 🔗 dzman How :)?
15:45 🔗 midas but there is not an easy way to search in the entire collection
15:45 🔗 midas add the url in the wayback machine
15:45 🔗 midas it will find it for you
15:46 🔗 dzman What can i use in the url?
15:47 🔗 dzman http://hyves.nl/username ?
15:47 🔗 midas the username if you know it
15:47 🔗 midas i think it was username.hyves.nl
15:47 🔗 dzman ah thanks, i will try
15:48 🔗 midas good luck :)
15:49 🔗 MMovie has quit IRC (Read error: Operation timed out)
15:51 🔗 MMovie has joined #archiveteam
15:51 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
15:52 🔗 dzman it says Page cannot be crawled or displayed due to robots.txt. every time i use my username :/
16:00 🔗 midas well damn
16:00 🔗 MMovie has quit IRC (Read error: Operation timed out)
16:00 🔗 midas hyves.nl redirects to hyvesgames.nl which doesnt allow the archive.org crawler
16:00 🔗 midas thats shitty.
16:01 🔗 midas joepie91: did you notice this yet?
16:01 🔗 MMovie has joined #archiveteam
16:04 🔗 dzman has quit IRC (Ping timeout: 255 seconds)
16:18 🔗 joepie91 I did not
16:20 🔗 PurpleSym I downloaded the CDX files for hyves.
16:20 🔗 PurpleSym And could grep them.
16:20 🔗 MMovie has quit IRC (Read error: Operation timed out)
16:21 🔗 MMovie has joined #archiveteam
16:24 🔗 metalcamp has quit IRC (Ping timeout: 258 seconds)
16:38 🔗 MMovie has quit IRC (Read error: Operation timed out)
16:40 🔗 MMovie has joined #archiveteam
16:51 🔗 bwn has quit IRC (Read error: Operation timed out)
16:54 🔗 atomotic has joined #archiveteam
16:56 🔗 MMovie has quit IRC (Read error: Operation timed out)
16:57 🔗 MMovie has joined #archiveteam
16:59 🔗 JesseW has joined #archiveteam
17:13 🔗 scyther has quit IRC (Quit: Leaving)
17:13 🔗 MMovie has quit IRC (Read error: Operation timed out)
17:14 🔗 MMovie has joined #archiveteam
17:22 🔗 JesseW has quit IRC (Quit: Leaving.)
17:31 🔗 MMovie has quit IRC (Read error: Operation timed out)
17:32 🔗 MMovie has joined #archiveteam
17:33 🔗 xXx_ndidd has joined #archiveteam
17:38 🔗 bwn has joined #archiveteam
17:46 🔗 ndizzle has quit IRC (Read error: Operation timed out)
17:47 🔗 MMovie has quit IRC (Read error: Operation timed out)
17:48 🔗 MMovie has joined #archiveteam
18:06 🔗 MMovie has quit IRC (Read error: Operation timed out)
18:07 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
18:08 🔗 MMovie has joined #archiveteam
18:13 🔗 vitzli has quit IRC (Leaving)
18:16 🔗 ndizzle has joined #archiveteam
18:29 🔗 xXx_ndidd has quit IRC (Read error: Operation timed out)
18:35 🔗 MMovie has quit IRC (Read error: Operation timed out)
18:36 🔗 MMovie has joined #archiveteam
18:45 🔗 HCross has quit IRC (Read error: Connection reset by peer)
18:49 🔗 HCross has joined #archiveteam
19:05 🔗 yipdw has quit IRC (Ping timeout: 1224 seconds)
19:05 🔗 signius has quit IRC (Ping timeout: 345 seconds)
19:05 🔗 FalconK has quit IRC (Ping timeout: 345 seconds)
19:05 🔗 hive-mind has quit IRC (Ping timeout: 316 seconds)
19:05 🔗 SirCmpwn has quit IRC (Ping timeout: 345 seconds)
19:05 🔗 bauruine has quit IRC (Ping timeout: 316 seconds)
19:05 🔗 ploopkaz- has quit IRC (Ping timeout: 345 seconds)
19:05 🔗 Atluxity has quit IRC (Ping timeout: 345 seconds)
19:05 🔗 balrog has quit IRC (Ping timeout: 345 seconds)
19:05 🔗 dan- has quit IRC (Ping timeout: 345 seconds)
19:05 🔗 HCross2 has quit IRC (Read error: Connection reset by peer)
19:05 🔗 johtso has quit IRC (Read error: Connection reset by peer)
19:05 🔗 zhongfu has quit IRC (Remote host closed the connection)
19:05 🔗 wp494_ has joined #archiveteam
19:05 🔗 victor has quit IRC (Write error: Broken pipe)
19:05 🔗 d_rebel has quit IRC (Write error: Connection reset by peer)
19:05 🔗 Vito` has quit IRC (Write error: Connection reset by peer)
19:05 🔗 winr4r has quit IRC (Write error: Connection reset by peer)
19:05 🔗 victor has joined #archiveteam
19:05 🔗 Vito` has joined #archiveteam
19:05 🔗 _desu____ has joined #archiveteam
19:05 🔗 hive-mind has joined #archiveteam
19:05 🔗 bauruine has joined #archiveteam
19:06 🔗 d_rebel has joined #archiveteam
19:06 🔗 Boltsie_ has joined #archiveteam
19:06 🔗 balrog has joined #archiveteam
19:06 🔗 swebb sets mode: +o balrog
19:06 🔗 ploopkazo has joined #archiveteam
19:06 🔗 deathy_ has joined #archiveteam
19:06 🔗 Atluxity has joined #archiveteam
19:06 🔗 TheKiwi_ has joined #archiveteam
19:06 🔗 Ungstein1 has joined #archiveteam
19:06 🔗 FalconK has joined #archiveteam
19:06 🔗 beeper_ has joined #archiveteam
19:07 🔗 casdr_ has joined #archiveteam
19:07 🔗 casdr_ has quit IRC (Connection closed)
19:07 🔗 beeper_ has quit IRC (Connection closed)
19:07 🔗 TheKiwi_ has quit IRC (Connection closed)
19:07 🔗 kevin_ has joined #archiveteam
19:07 🔗 casdr_ has joined #archiveteam
19:07 🔗 TheKiwi_ has joined #archiveteam
19:07 🔗 beeper_ has joined #archiveteam
19:07 🔗 zhongfu has joined #archiveteam
19:08 🔗 Ungstein has quit IRC (Ping timeout: 274 seconds)
19:08 🔗 wp494 has quit IRC (Ping timeout: 274 seconds)
19:08 🔗 _desu___ has quit IRC (Ping timeout: 274 seconds)
19:08 🔗 Boltsie has quit IRC (Ping timeout: 274 seconds)
19:08 🔗 JSharp___ has quit IRC (Ping timeout: 274 seconds)
19:08 🔗 TheKiwi has quit IRC (Ping timeout: 274 seconds)
19:08 🔗 Ctrl-S___ has quit IRC (Ping timeout: 274 seconds)
19:08 🔗 beeper has quit IRC (Ping timeout: 274 seconds)
19:08 🔗 kevin has quit IRC (Ping timeout: 274 seconds)
19:08 🔗 casdr has quit IRC (Ping timeout: 274 seconds)
19:08 🔗 abartov__ has quit IRC (Ping timeout: 274 seconds)
19:08 🔗 karissa__ has quit IRC (Ping timeout: 274 seconds)
19:08 🔗 VonGuard has quit IRC (Ping timeout: 274 seconds)
19:08 🔗 deathy has quit IRC (Ping timeout: 274 seconds)
19:08 🔗 _desu____ is now known as _desu___
19:08 🔗 Boltsie_ is now known as Boltsie
19:08 🔗 deathy_ is now known as deathy
19:08 🔗 SirCmpwn has joined #archiveteam
19:08 🔗 casdr_ is now known as casdr
19:08 🔗 beeper_ is now known as beeper
19:08 🔗 TheKiwi_ is now known as TheKiwi
19:08 🔗 kevin_ is now known as kevin
19:08 🔗 abartov__ has joined #archiveteam
19:08 🔗 dan- has joined #archiveteam
19:08 🔗 HCross2 has joined #archiveteam
19:09 🔗 beeper has quit IRC (Remote host closed the connection)
19:09 🔗 beeper has joined #archiveteam
19:10 🔗 johtso has joined #archiveteam
19:10 🔗 signius has joined #archiveteam
19:10 🔗 TheKiwi has quit IRC (Remote host closed the connection)
19:11 🔗 TheKiwi has joined #archiveteam
19:12 🔗 winr4r has joined #archiveteam
19:23 🔗 jut has quit IRC (jut)
19:30 🔗 Tomcat_ has joined #archiveteam
19:33 🔗 Tomcat__ has joined #archiveteam
19:35 🔗 Tomcat_ has quit IRC (Read error: Operation timed out)
19:56 🔗 FalconK Fletcher: code should drop this week
20:05 🔗 MMovie has quit IRC (Read error: Operation timed out)
20:05 🔗 Tomcat__ has quit IRC (Remote host closed the connection)
20:06 🔗 MMovie has joined #archiveteam
20:22 🔗 MMovie has quit IRC (Read error: Operation timed out)
20:24 🔗 MMovie has joined #archiveteam
20:29 🔗 ndiddy has joined #archiveteam
20:30 🔗 ndizzle has quit IRC (Read error: Operation timed out)
20:39 🔗 dashcloud has quit IRC (Read error: Operation timed out)
20:42 🔗 dashcloud has joined #archiveteam
20:49 🔗 MMovie has quit IRC (Read error: Operation timed out)
20:51 🔗 MMovie has joined #archiveteam
21:01 🔗 metalcamp has joined #archiveteam
21:06 🔗 MMovie has quit IRC (Read error: Operation timed out)
21:08 🔗 MMovie has joined #archiveteam
21:20 🔗 jake1 has joined #archiveteam
21:21 🔗 MMovie has quit IRC (Read error: Operation timed out)
21:23 🔗 MMovie has joined #archiveteam
21:24 🔗 jake1 MrRadar: SketchCow passed along the OverflowError bug you found in the ia CLI when uploading files larger than ~2GB. This should fix it here: https://github.com/jjjake/internetarchive/commit/6c9f77cb9b57296bc88278b5716c7a3bc32c3b43
21:25 🔗 jake1 That fix will be in v1.0.2, which I hope to release later today.
21:33 🔗 schbirid has quit IRC (Quit: Leaving)
21:34 🔗 * JW_work waves to jake1 :-)
21:46 🔗 MMovie has quit IRC (Read error: Operation timed out)
21:48 🔗 MMovie has joined #archiveteam
21:56 🔗 fie has joined #archiveteam
22:01 🔗 dxrt Thanks jake1!
22:07 🔗 dashcloud has quit IRC (Read error: Operation timed out)
22:08 🔗 MrRadar jake1: Thanks for the fix.
22:09 🔗 metalcamp has quit IRC (Ping timeout: 258 seconds)
22:11 🔗 dashcloud has joined #archiveteam
22:24 🔗 MMovie has quit IRC (Read error: Operation timed out)
22:25 🔗 MMovie has joined #archiveteam
22:42 🔗 MMovie has quit IRC (Read error: Operation timed out)
22:43 🔗 MMovie has joined #archiveteam
22:47 🔗 ErkDog OK so I told roninski I would help him get a folder out of a web archive, so I downloaded it and extracted it, so now I have this 138 Gig warc file that I don't know what to do with :(
22:49 🔗 MrRadar You can use warcat to extract it
22:49 🔗 MrRadar https://pypi.python.org/pypi/Warcat/
22:49 🔗 MrRadar There are other tools as well
22:49 🔗 ErkDog that's pythony, I'm in windows :(
22:51 🔗 yipdw has joined #archiveteam
22:52 🔗 ErkDog this looks promising: https://github.com/ikreymer/webarchiveplayer
22:52 🔗 MrRadar Well, that will let you browse the web archive (like the IA's Wayback Machine)
22:52 🔗 MrRadar You can install Python on Windows
22:54 🔗 wp494_ is now known as wp494
22:55 🔗 ErkDog ohhhh yeah I guess I can
22:56 🔗 ErkDog lol it would be cool if I could run the scripts from windows command line instead of loosing all these resources to the virtual box hypervisor, lol
22:57 🔗 JW_work ErkDog: and actually, there's an instance of (a variant of) webarchiveplayer running at http://archivelab.org:3579/item/{IA identifier} so if you put the archive identifier you downloaded in at the end, then /*/ then the website, you should be able to get it without even downloading it.
22:59 🔗 JW_work has left
22:59 🔗 JW_work has joined #archiveteam
22:59 🔗 ErkDog OOOO'rly thanks JW_work
22:59 🔗 MMovie has quit IRC (Read error: Operation timed out)
22:59 🔗 yipdw so, webarchiveplayer is also bundled as a Windows application
23:01 🔗 MMovie has joined #archiveteam
23:01 🔗 ErkDog what constitutes the "IA Identifier" of : https://archive.org/download/archiveteam-fanfiction-warc-09
23:01 🔗 RedType has left
23:02 🔗 MrRadar archiveteam-fanfiction-warc-09
23:17 🔗 VADemon has quit IRC (Quit: left4dead)
23:19 🔗 MMovie has quit IRC (Read error: Operation timed out)
23:21 🔗 MMovie has joined #archiveteam
23:38 🔗 MMovie has quit IRC (Read error: Operation timed out)
23:40 🔗 MMovie has joined #archiveteam
23:56 🔗 MMovie has quit IRC (Read error: Operation timed out)
23:57 🔗 MMovie has joined #archiveteam
irclogger-viewer