[00:26] *** Ymgve has quit IRC () [00:33] *** Start is now known as StartAway [00:33] *** schbirid2 has joined #archiveteam [00:35] *** schbirid has quit IRC (Read error: Operation timed out) [00:44] *** wp494 has joined #archiveteam [00:44] *** BiggieJo1 has joined #archiveteam [00:47] *** BiggieJon has quit IRC (Read error: Operation timed out) [00:53] *** Mayonaise has quit IRC (Ping timeout: 365 seconds) [00:53] *** zenguy_pc has quit IRC (Read error: Operation timed out) [01:06] *** primus104 has quit IRC (Leaving.) [01:09] *** zenguy_pc has joined #archiveteam [01:38] *** Mayonaise has joined #archiveteam [01:40] *** StartAway is now known as Start [01:57] *** xk_id has quit IRC (Ping timeout: 620 seconds) [02:06] *** Mayonaise has quit IRC (Ping timeout: 365 seconds) [02:11] *** Mayonaise has joined #archiveteam [02:35] *** logchfoo starts logging #archiveteam at Mon Nov 10 02:35:30 2014 [02:35] *** logchfoo has joined #archiveteam [02:40] *** ex-parrot has joined #archiveteam [02:52] *** Ravenloft has quit IRC (Ping timeout: 606 seconds) [02:57] *** bauruine has quit IRC (Ping timeout: 265 seconds) [03:03] *** schbirid2 has quit IRC (Read error: Operation timed out) [03:03] *** bauruine has joined #archiveteam [03:11] *** schbirid2 has joined #archiveteam [03:47] *** mistym has joined #archiveteam [03:51] *** Lord_Nigh has quit IRC (Read error: Operation timed out) [03:53] *** Lord_Nigh has joined #archiveteam [04:23] *** bsmith093 has quit IRC (Read error: Operation timed out) [04:38] *** mistym has quit IRC (Remote host closed the connection) [04:39] *** bsmith093 has joined #archiveteam [04:53] *** ex-parrot has quit IRC (Leaving.) [04:56] *** aaaaaaaaa has quit IRC (Leaving) [05:23] *** mistym has joined #archiveteam [05:33] *** JonimusP is now known as Jonimus [06:08] *** Start is now known as StartAway [06:08] *** signius has quit IRC (Ping timeout: 480 seconds) [06:18] *** signius has joined #archiveteam [07:09] *** ruukasu has quit IRC (Quit: WeeChat 1.0.1) [07:14] *** ruukasu has joined #archiveteam [07:20] arkiver: chfoo: Is SWIPNET data done being uploaded, or is that still happening? [07:23] *** primus104 has joined #archiveteam [07:39] *** rduser has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** SadDM has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** twrist has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** matthusb_ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** altlabel has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** pikhq has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** ionpulse has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Jogie has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** everdred has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** eprillios has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Insomnia_ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** gibigiana has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Aranje has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** dcmorton has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** SmileyG has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Cameron_D has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** slash` has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** pft has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Sanqui has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** antomatic has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** khaoohs has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Sue_ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** mistym has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Lord_Nigh has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** GLaDOS has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** arkiver has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Shank_ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** RainbowCo has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** fluff has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** lhobas has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** blergh has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Nemo_bis has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Riviera has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** SN4T14__ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** ats has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** aNthraXx has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** brayden has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Zebranky_ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Deewiant_ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Atluxity has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** tfgbd has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** namespace has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** ruukasu has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** bauruine has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** pfallenop has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Sellyme_ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** dashcloud has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Elegance has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** PepsiMax has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** tephra_ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** DFJustin has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Sk1d has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** NovaKing_ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** zifnab06 has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** danneh_ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** soultcer has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Kniffy has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** primus104 has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** schbirid2 has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Coderjoe has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** norbert79 has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** is- has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** nox has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** ersi has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** ZorbaBeta has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** garyrh has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** will__ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Void_ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Boppen has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Kenshin has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** wp494 has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** trs80 has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** filippo_ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** mhazinsk has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** bsmith093 has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** SketchCow has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** kanzure has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** lytv has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** RedType has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** ryan___ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** chfoo has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** dx has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** xmc has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** zenguy_pc has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** BlueMaxim has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** StartAway has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** primus_ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Jonimus has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Meeh_ has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** swebb has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** lemonkey has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** w0rp has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** RealMarc has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** lukeman has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** dserodio has quit IRC (ircd.choopa.net hub.efnet.us) [07:39] *** Laverne has quit IRC (ircd.choopa.net hub.efnet.us) [17:03] *** logchfoo starts logging #archiveteam at Mon Nov 10 17:03:04 2014 [17:03] *** logchfoo has joined #archiveteam [17:03] *** vman315 has joined #archiveteam [17:04] *** vman315 has quit IRC (Client Quit) [17:05] *** nertzy has quit IRC (This computer has gone to sleep) [17:06] *** average has joined #archiveteam [17:06] hello archiveteam [17:07] i've been watching your activities [17:07] i am proud of you [17:07] you're amazing [17:07] * schbirid2 fetches the glitter box [17:08] may I ask a question related to the Twitter stream dump ? [17:08] for example this one https://archive.org/details/archiveteam-twitter-stream-2014-09 [17:08] (btw I'm from Romania, so I'm not an english speaker, please excuse my grammar or spelling errors) [17:10] the Twitter stream dump. uhm, how is it produced ? [17:10] I mean, how do you consume it ? I've been able to use it as well [17:10] arkiver SketchCow: swipnet should be done. it hasn't been in the tracker for a while. [17:10] with the api endpoint https://stream.twitter.com/1.1/statuses/filter.json [17:11] and the parameters delimited=length&track=romania [17:11] I've seen that archiveteam has a github account with all the code that goes into pulling this data [17:11] I was unable to find the code for the Twitter stream dumper [17:11] could anyone please point me to it ? [17:12] i don't think we are able to share that [17:12] can you talk about it then ? [17:12] if sharing is not an option [17:13] SmileyG: ^^ [17:13] hmmmm i think we can talk about it - however i personally don't know much about it [17:13] other than we have some special access to something [17:14] ah I see. so basically if I were to try to do what you're doing, I wouldn't be able to, right ? [17:14] espes__: if i recall correctly, the scripts paginated everything on the homepage including friends list. but that's all saved in the warcs and difficult to get back out. [17:14] *** mistym has quit IRC (Remote host closed the connection) [17:15] average: yeah pretty much :( [17:16] SmileyG: does this mean you have access to stream.twitter.com/1.1/statuses/firehose.json ? [17:16] SmileyG: aka firehose (https://dev.twitter.com/streaming/firehose) [17:16] I don't think we have full access no [17:16] I see [17:16] i believe we get it via somewhere else. [17:16] .... okay, wow [17:16] but they do, or something [17:16] so I accidentally ended up on the Hyves Games forums [17:16] * SmileyG isn't sure. This is all speculation. [17:16] "to keep the forums clean, this topic will be locked and eventually removed" [17:16] DO THESE PEOPLE NEVER FUCKING LEARN [17:17] https://archive.org/details/twitterstream [17:17] schbirid2: yes, I know about that [17:17] is this the first time an archiveteam target is deleting data a *second* time? [17:17] average: it says spritzer right there [17:17] cc midas [17:17] schbirid2: ah ! where's spritzer ? [17:18] SmileyG: you guys, like all the archiveteam, are a hidden jewel of the interwebz [17:18] I swear [17:18] right, gotta go. [17:19] ok [17:25] *** Trskl has joined #archiveteam [17:26] is there a api/tool to pull a specific file from a warc? [17:27] espes__: there's a bunch, I don't think any of them support doing it directly from IA though [17:28] Hi all. I didn't see a wiki page and was wondering if anyone had considered archiving Yahoo Directory before it shuts down in December. [17:29] (If you'll recall, before Yahoo did the search engine thing, the manually-curated directory *was* the original Yahoo product.) [17:29] It's set to disappear on 12/31. http://yahoo.tumblr.com/post/98474044364/progress-report-continued-product-focus [17:31] *** logchfoo starts logging #archiveteam at Mon Nov 10 17:31:59 2014 [17:31] *** logchfoo has joined #archiveteam [17:32] *** mistym has joined #archiveteam [17:49] I think we've already crawled it.... [17:51] They had some server problems on the first few attempts. I don't know if they tried lately. [17:51] *** primus104 has quit IRC (Leaving.) [17:53] Wow, that was unclear. Yahoo was having server problems the first few times it was tried. I don't know if those were fixed or if there have been any recent attempts to get it. [17:54] D: [17:54] so basically we broke yahoo? [17:56] *** bobby_ has joined #archiveteam [18:00] *** kahayes has quit IRC (Read error: Connection reset by peer) [18:04] I see. Basically, just wanted to come by and shine some light on Directory in case its closing hadn't been noticed before. I don't really have the skill set to troubleshoot; I'm more of a random sympathetic party who tries to run a seesaw script now and then to help out. :) [18:05] I'll trust that more capable minds can figure out whether it's doable. [18:10] So my Dark Horse forum grab finished, but it looks like it's probably incomplete. [18:10] http://boards.darkhorse.com/ indicates that there are 17491 topics, which means that there should be *at least* that many urls fetched... [18:10] but the phpbb3 lua script says that it grabbed less than 10350 [18:15] everdred: all help is very welcome :D [18:17] Internet Archive inside is running to an Agile model. [18:17] That should be.... interesting. [18:17] zcat boards.darkhorse.com-20141110.warc.gz | grep 'WARC-Type: request'|wc indicates that there are 10158 files in the warc :-( [18:17] About to find out my role in a "Content Scrum" [18:20] how do I rename a file if it isn't showing up in the file editor? [18:21] I just uploaded extra stuff with the python uploader and they aren't showing up in the list :( [18:23] *** bobby__ has joined #archiveteam [18:25] SketchCow: oh dear [18:25] :) [18:25] well to be fair, the ideas behind agile can work really well... just most organizations/people don't really seem to implement them correctly [18:26] *** bobby_ has quit IRC (Ping timeout: 240 seconds) [18:26] let's hope that works better here [18:26] * joepie91 resumes writing hyves parsing code [18:38] tfgbd: if you just uploaded it, you might have to wait for the jobs to finish running [18:56] joepie91> is this the first time an archiveteam target is deleting data a *second* time? <-- i would find it hard to believe that it would have taken this long. [19:05] *** primus104 has joined #archiveteam [19:07] xmc: same, but I can't think of any prior instance... [19:08] i'm almost certain there was something about a year ago [19:08] but i can't remember more detailedly than that [19:42] *** Aranje has quit IRC (Quit: Three sheets to the wind) [20:06] *** ex-parrot has joined #archiveteam [20:08] *** ex-parrot has quit IRC (Client Quit) [20:19] *** nertzy has joined #archiveteam [20:23] *** Panasonic has quit IRC (Ping timeout: 480 seconds) [20:24] *** bobby__ has quit IRC (Ping timeout: 240 seconds) [20:30] *** bobby_ has joined #archiveteam [21:01] *** nertzy has quit IRC (This computer has gone to sleep) [21:03] chfoo: you're keeping some great logs with logchfoo [21:03] Are you thinking about grabbing more channels? [21:09] *** bobby_ has quit IRC (Quit: Page closed) [21:10] *** Bobby_ has joined #archiveteam [21:28] *** BlueMaxim has joined #archiveteam [21:31] *** mistym has quit IRC (Remote host closed the connection) [21:44] arkiver: what other channels do you have in mind? [21:46] *** Bobby_ has quit IRC () [21:52] *** mistym has joined #archiveteam [21:53] what's the link to your archives, chfoo? [21:53] i'm bad at website. [21:53] http://archive.fart.website/bin/irclogger_logs [21:54] thx [21:55] i see #quitpic , #oodbye missing [21:55] i'm not in any other not-in-fartwebsite project channels atm [21:56] but i think all the project channels should be logged [21:57] agreed [22:05] chfoo: I think it would be great if you are able to log all the channels [22:12] joins may be considered unnecessarily noisy [22:22] *** mistym has quit IRC (Remote host closed the connection) [22:37] *** mistym has joined #archiveteam [22:54] *** RedType has quit IRC (Quit: leaving) [22:54] *** RedType has joined #archiveteam [22:54] *** RedType has quit IRC (Client Quit) [23:07] *** RedType has joined #archiveteam [23:12] *** Ymgve__ has joined #archiveteam [23:15] *** nertzy has joined #archiveteam [23:16] *** Ymgve has quit IRC (Ping timeout: 512 seconds) [23:36] *** nertzy has quit IRC (This computer has gone to sleep)