[00:05] *** lelo_paul has quit IRC (Read error: Connection reset by peer) [00:05] *** paul_lelo has joined #archiveteam [00:11] *** Froggypwn has quit IRC (Read error: Connection reset by peer) [00:12] *** Fletcher has quit IRC (Read error: Operation timed out) [00:13] *** Froggypwn has joined #archiveteam [00:13] *** Jordan has quit IRC (Ping timeout: 250 seconds) [00:16] *** Jordan has joined #archiveteam [00:20] *** icedice has quit IRC (Quit: Leaving) [00:21] *** Fletcher has joined #archiveteam [00:21] *** SmileyG has joined #archiveteam [00:22] *** Smiley has quit IRC (Read error: Operation timed out) [00:23] *** hawc145 has joined #archiveteam [00:25] *** sivoais has quit IRC (Remote host closed the connection) [00:26] *** sivoais has joined #archiveteam [00:26] *** HCross has quit IRC (Ping timeout: 370 seconds) [00:32] *** WinterFox has joined #archiveteam [00:39] *** BitHippo has quit IRC (Ping timeout: 268 seconds) [00:40] *** dashcloud has quit IRC (Ping timeout: 244 seconds) [00:42] *** dashcloud has joined #archiveteam [00:45] *** vitzli has joined #archiveteam [00:58] *** voltagex has joined #archiveteam [00:59] https://news.ycombinator.com/item?id=12805415 [01:01] cc arkiver ^ [01:01] something for the PDF downloading project? [01:01] voltagex: ref http://archiveteam.org/index.php?title=PDF_2016 [01:02] It's a Big Deal but quite niche. [01:02] Happy to throw some AWS resources at it, but I can't do much coding this weekend [01:03] Plus I find Warrior and the related Python stuff confusing as hell :p [01:03] voltagex: don't know if it'd warrant a separate project [01:03] voltagex: where would one get said PDFs? [01:03] hm, PDF project is flatlined currently [01:04] No idea, I'm not in that industry. Just relaying the bat signal :) [01:06] Ah, see you're on the thread. I better get back to work. Thanks [01:07] yep :) [01:07] voltagex: thanks for the notice [01:09] *** Specular has joined #archiveteam [01:11] *** BartoCH has quit IRC (Ping timeout: 260 seconds) [01:18] *** BartoCH has joined #archiveteam [01:24] *** JesseW has quit IRC (Ping timeout: 370 seconds) [01:41] *** Specular has quit IRC (Ping timeout: 370 seconds) [01:43] *** Specular has joined #archiveteam [02:09] *** zenguy has quit IRC (Read error: Operation timed out) [02:32] *** zenguy has joined #archiveteam [03:18] *** pikhq has quit IRC (Ping timeout: 255 seconds) [03:19] *** pikhq has joined #archiveteam [03:19] *** Start has quit IRC (Read error: Connection reset by peer) [03:20] *** Start has joined #archiveteam [03:32] *** maelstrom has quit IRC (Remote host closed the connection) [03:41] *** SDr has joined #archiveteam [03:41] *** Stiletto has joined #archiveteam [03:41] *sigh* [03:42] Vine: DEFCON 2 [03:43] friend of mine is wondering, if #archiveteam is even within an order-of-mag close in capacity to do vine? [03:47] SDr: #vinewhine [04:12] *** ndiddy has quit IRC (Read error: Connection reset by peer) [04:13] *** anan has joined #archiveteam [04:13] *** anan has quit IRC (Client Quit) [04:17] https://medium.com/@vine/important-news-about-vine-909c5f4ae7a7#.5dj3qhybi [04:20] *** vitzli has quit IRC (Quit: Leaving) [04:21] *** Sk1d has quit IRC (Ping timeout: 194 seconds) [04:27] *** Sk1d has joined #archiveteam [04:37] *** somerando has joined #archiveteam [04:39] *** somerando has quit IRC (Client Quit) [04:45] *** Specular has quit IRC (Ping timeout: 370 seconds) [04:46] *** Specular has joined #archiveteam [05:30] *** RichardG has joined #archiveteam [05:43] *** Specular_ has joined #archiveteam [05:46] *** Specular has quit IRC (Ping timeout: 370 seconds) [05:52] https://ia801508.us.archive.org/8/items/ia20thanniversaryevent/ia20thanniversaryevent_720p.mp4 [06:02] Ignore the no sound at the beginning, no idea about that [06:04] *** Temp__ has joined #archiveteam [06:05] *** Temp__ has left [06:07] *** wp494 has quit IRC (Read error: Operation timed out) [06:11] *** superkuh has quit IRC (Quit: the neuronal action potential is an electrical manipulation of reversible abrupt phase changes in the lipid bilaye) [06:16] *** wp494 has joined #archiveteam [06:25] *** superkuh has joined #archiveteam [06:27] *** JesseW has joined #archiveteam [06:33] *** superkuh has quit IRC (Quit: the neuronal action potential is an electrical manipulation of reversible abrupt phase changes in the lipid bilaye) [06:35] *** Aranje has quit IRC (Quit: Three sheets to the wind) [06:39] *** superkuh has joined #archiveteam [06:52] *** superkuh has quit IRC (Quit: the neuronal action potential is an electrical manipulation of reversible abrupt phase changes in the lipid bilaye) [06:54] *** superkuh has joined #archiveteam [06:55] *** RichardG_ has joined #archiveteam [06:55] *** RichardG has quit IRC (Read error: Connection reset by peer) [07:01] *** RichardG_ has quit IRC (Ping timeout: 244 seconds) [07:01] *** RichardG has joined #archiveteam [07:01] *** closure has quit IRC (Ping timeout: 244 seconds) [07:01] *** nekomune has quit IRC (Ping timeout: 244 seconds) [07:02] *** fie has quit IRC (Ping timeout: 244 seconds) [07:02] *** Frogging has quit IRC (Ping timeout: 244 seconds) [07:02] *** espes__ has quit IRC (Ping timeout: 244 seconds) [07:03] *** edsu has quit IRC (Ping timeout: 244 seconds) [07:03] *** alfiepate has quit IRC (Ping timeout: 244 seconds) [07:04] *** closure has joined #archiveteam [07:04] *** Frogging has joined #archiveteam [07:04] *** JesseW has quit IRC (Ping timeout: 370 seconds) [07:05] *** jk[SVP] has quit IRC (Ping timeout: 244 seconds) [07:05] *** alfie has joined #archiveteam [07:06] *** jk[SVP] has joined #archiveteam [07:06] *** nekomune has joined #archiveteam [07:06] https://vine.co/v/Oetanigr7i6 [07:07] *** Mathias` has quit IRC (Ping timeout: 244 seconds) [07:07] *** Baljem has quit IRC (Ping timeout: 244 seconds) [07:08] *** espes__ has joined #archiveteam [07:08] *** RichardG has quit IRC (Read error: Operation timed out) [07:08] *** Madthias has joined #archiveteam [07:09] *** closure has quit IRC (Ping timeout: 244 seconds) [07:09] *** superkuh has quit IRC (Remote host closed the connection) [07:11] *** nox_ has quit IRC (Ping timeout: 244 seconds) [07:11] *** edsu has joined #archiveteam [07:11] *** swebb sets mode: +o edsu [07:12] *** chfoo has quit IRC (Read error: Operation timed out) [07:12] *** chfoo has joined #archiveteam [07:13] *** closure has joined #archiveteam [07:13] *** Baljem has joined #archiveteam [07:14] *** SilSte has quit IRC (Read error: Operation timed out) [07:15] *** RichardG has joined #archiveteam [07:15] *** Fletcher has quit IRC (Read error: Operation timed out) [07:16] *** espes___ has joined #archiveteam [07:17] *** Fletcher has joined #archiveteam [07:17] *** Whopper has joined #archiveteam [07:17] *** kristian_ has joined #archiveteam [07:18] *** espes__ has quit IRC (Read error: Connection reset by peer) [07:18] *** JW_work1 has joined #archiveteam [07:18] *** JW_work has quit IRC (Read error: Connection reset by peer) [07:21] *** TC02 has quit IRC (Ping timeout: 633 seconds) [07:21] *** TC02 has joined #archiveteam [07:22] *** TC01 has quit IRC (Ping timeout: 633 seconds) [07:22] *** Whopper_ has quit IRC (Ping timeout: 633 seconds) [07:22] *** maseck has quit IRC (Ping timeout: 633 seconds) [07:22] *** yuitimoth has quit IRC (Ping timeout: 633 seconds) [07:23] *** brayden_ has quit IRC (Ping timeout: 633 seconds) [07:24] *** TC01 has joined #archiveteam [07:25] *** SilSte has joined #archiveteam [07:29] *** maseck has joined #archiveteam [07:29] *** yuitimoth has joined #archiveteam [07:35] *** nox_ has joined #archiveteam [07:41] *** superkuh has joined #archiveteam [08:02] *** superkuh has quit IRC (Quit: the neuronal action potential is an electrical manipulation of reversible abrupt phase changes in the lipid bilaye) [08:05] Official vine for "The Archive Team is on it": https://vine.co/v/ijKqnJqbjlv [08:11] *** superkuh has joined #archiveteam [08:13] Again, someone uploaded 45gb of TV rips [08:15] It has a 95% chance of going into the trash [08:15] (I mean to the FTP site) [08:16] Going to be sad when I change the password to reflect I need to revisit who's connecting [08:22] I went through it [08:22] Crocodile Dundee, DVD RIP, is no way to go through life, son [08:58] Anyway. More relevant is that FOS is still choking to death on Orkut. I just ran parallel moves to move Yahoo Answers and Panormaio uploads into the rest of the pipeline, since the machine has been focused on Orkut forever and a day. [08:58] in the future we need to engineer these uploads so they don't cause millions of small warc.gz files, even if it means some weird work. The megawarc chokes it [09:00] (Archivebot is above all this crap - different pipeline, goes frequently) [09:00] don't move the toontastic folder [09:01] i have like still have +100gb of toontastic [09:01] of course i don't know if you would still want it [09:02] we sort of got enough of it show examples of it [09:13] *** ravetcofx has quit IRC (Ping timeout: 506 seconds) [09:29] *** BlueMaxim has quit IRC (Quit: Leaving) [09:43] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [09:44] *** SilSte has joined #archiveteam [09:48] *** Cameron_D has quit IRC (Ping timeout: 370 seconds) [09:52] *** xmc has quit IRC (Read error: Operation timed out) [09:52] *** RichardG has quit IRC (Ping timeout: 370 seconds) [09:53] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [09:53] *** SilSte has joined #archiveteam [09:55] *** xmc has joined #archiveteam [09:55] *** swebb sets mode: +o xmc [09:56] *** Cameron_D has joined #archiveteam [09:58] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [09:59] *** SilSte has joined #archiveteam [10:03] *** Specular_ has quit IRC (Quit: Leaving) [10:08] *** gibigiana has quit IRC (Ping timeout: 250 seconds) [10:09] *** gibigiana has joined #archiveteam [10:10] Maybe there's some easy optimisation possible? https://github.com/alard/megawarc/blob/master/megawarc#L272 [10:10] "seeking" a tar is hell [10:11] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [10:11] *** SilSte has joined #archiveteam [10:13] From a quick look I don't see if it reads the entire tar from scratch for each file contained [10:16] *** VADemon has joined #archiveteam [10:17] *** vineguy has joined #archiveteam [10:17] hi [10:21] *** vineguy has quit IRC (Client Quit) [10:21] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [10:21] *** SilSte has joined #archiveteam [10:29] Nemo_bis: Why do you think it is slow? What actions seems to be slow? Also I/O, CPU? ... What you pointed at is merely setting pointer of file cursor (unless the file is compressed, which in this case seems not), so that should be very, very cheap operation. [10:33] It has been a while since I looked at the code, so I know there are parts that could really be optimized a lot, but what you pointed at is not the case. Though the copying things around memory, processing of gz files, etc could be optimized vastly. Let me know please, interested in helping with this one. [10:33] *** SilSte has quit IRC (Remote host closed the connection) [10:34] *** SilSte has joined #archiveteam [11:09] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [11:12] *** SilSte has joined #archiveteam [11:15] *** Atom has quit IRC (Read error: Connection reset by peer) [11:19] *** brayden_ has joined #archiveteam [11:19] *** swebb sets mode: +o brayden_ [11:23] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [11:25] *** SilSte has joined #archiveteam [11:38] *** kristian_ has quit IRC (Quit: Leaving) [11:38] *** gibigiana has quit IRC (Ping timeout: 255 seconds) [11:40] *** gibigiana has joined #archiveteam [11:50] *** SilSte has quit IRC (Remote host closed the connection) [11:51] *** SilSte has joined #archiveteam [11:55] *** SilSte has quit IRC (Client Quit) [11:59] *** macker has left Leaving [12:01] *** SilSte has joined #archiveteam [12:04] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [12:04] *** SilSte has joined #archiveteam [12:09] *** Zialus has quit IRC (Read error: Operation timed out) [12:11] *** Zialus has joined #archiveteam [12:16] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [12:17] *** SilSte has joined #archiveteam [12:25] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [12:26] *** SilSte has joined #archiveteam [12:27] *** VADemon has quit IRC (Quit: left4dead) [12:33] *** incant has joined #archiveteam [12:40] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [12:45] *** SilSte has joined #archiveteam [12:48] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [12:49] *** SilSte has joined #archiveteam [12:53] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [12:54] *** SilSte has joined #archiveteam [12:57] *** SilSte has quit IRC (Client Quit) [13:01] *** SilSte has joined #archiveteam [13:16] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [13:17] *** SilSte has joined #archiveteam [13:19] *** Yoshimura has quit IRC (Remote host closed the connection) [13:22] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [13:23] *** SilSte has joined #archiveteam [13:26] *** Atom has joined #archiveteam [13:28] *** SilSte has quit IRC (Client Quit) [13:40] *** SilSte has joined #archiveteam [13:54] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [13:54] *** SilSte has joined #archiveteam [14:03] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [14:04] *** RichardG has joined #archiveteam [14:04] *** SilSte has joined #archiveteam [14:08] *** SilSte has quit IRC (Client Quit) [14:09] *** SilSte has joined #archiveteam [14:23] *** Start has quit IRC (Quit: Disconnected.) [14:27] *** WinterFox has quit IRC (Read error: Operation timed out) [14:43] *** RichardG has quit IRC (Ping timeout: 370 seconds) [14:55] *** Yoshimura has joined #archiveteam [14:56] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [14:58] *** SilSte has joined #archiveteam [15:03] *** Froggypwn has quit IRC (Read error: Connection reset by peer) [15:03] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [15:05] *** SilSte has joined #archiveteam [15:12] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [15:12] *** SilSte has joined #archiveteam [15:16] *** SilSte has quit IRC (Client Quit) [15:16] *** SilSte has joined #archiveteam [15:19] *** JesseW has joined #archiveteam [15:22] *** bithippo has joined #archiveteam [15:23] If one were to upload a WARC and CDX of a site rip from their own instance of ArchiveBot to the Internet Archive, what tags or other metadata should be included for someone to one day find it and include it in the Wayback Machine? [15:28] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [15:29] *** SilSte has joined #archiveteam [15:41] *** kristian_ has joined #archiveteam [15:49] *** SilSte has quit IRC (Remote host closed the connection) [15:50] *** SilSte has joined #archiveteam [15:54] *** SilSte has quit IRC (Remote host closed the connection) [15:55] *** SilSte has joined #archiveteam [16:01] *** SilSte has quit IRC (Quit: No Ping reply in 180 seconds.) [16:09] *** bsmith093 has quit IRC (Ping timeout: 255 seconds) [16:11] *** AlexLehm has joined #archiveteam [16:16] *** db48x has quit IRC (Read error: Connection reset by peer) [16:17] *** db48x has joined #archiveteam [16:32] *** RichardG has joined #archiveteam [16:34] *** JesseW has quit IRC (Ping timeout: 370 seconds) [16:49] *** kristian_ has quit IRC (Quit: Leaving) [16:50] bithippo: I'd at least add a 'warc' tag, but you need to ask an IA admin to add it to the wayback machine [16:50] bithippo: which roughly translates to "ask SketchCow" [17:07] joepie91: Thanks! [17:29] *** Stilett0 has joined #archiveteam [17:29] *** superkuh has quit IRC (Quit: the neuronal action potential is an electrical manipulation of reversible abrupt phase changes in the lipid bilaye) [17:31] *** Stiletto has quit IRC (Read error: Operation timed out) [17:56] *** RichardG has quit IRC (Read error: Operation timed out) [17:56] *** RichardG has joined #archiveteam [18:00] *** Start has joined #archiveteam [18:04] *** bithippo has quit IRC (Ping timeout: 268 seconds) [18:20] *** Start has quit IRC (Quit: Disconnected.) [18:24] *** ndiddy has joined #archiveteam [18:30] *** ravetcofx has joined #archiveteam [18:32] Hm, probably worth (at *very* low priority) grabbing https://lists.gnu.org/archive/html [18:36] *** Start has joined #archiveteam [18:37] *** incant has quit IRC (Read error: Operation timed out) [18:53] *** bsmith093 has joined #archiveteam [18:57] *** DragonDav has joined #archiveteam [18:57] *** Start has quit IRC (Quit: Disconnected.) [19:16] As someone who's spent a few years of their life scraping websites for a living (in Python), I'm trying to see how I might make myself useful. Not really getting a distinct thing I can make a difference with on the website. [19:16] *** DragonDav is now known as Dragon [19:22] *** Froggypwn has joined #archiveteam [19:25] If anyone wants to get in touch with me to suggest something I can usefully do, dave.mckee@gmail.com [19:26] Dragon: have you had a look at http://archiveteam.org/index.php?title=Dev ? [19:30] That seems very much 'how to' -- which is excellent -- but more looking for a "what" to tackle [19:32] Not too sure if there's a todo list of sorts, but I'm sure plenty of people have ideas with what can be improved. Guess that's probably something we could have on the wiki [19:33] I'd guess Vine us something that would need to be tackled - although I don't know what other people have done so far regarding it [19:33] Dragon: #archiveteam-bs to talk? or put up skill list somewhere [20:02] *** JW_work has joined #archiveteam [20:05] *** JW_work1 has quit IRC (Read error: Operation timed out) [20:06] Dragon: URLTeam could use someone who wanted to work on figuring out how to scrape resistant or unusual URL shortening services. [20:07] Join #urlteam and ask me there [20:11] *** JW_work has quit IRC (Quit: Leaving.) [20:13] *** JW_work has joined #archiveteam [20:27] *** AlexLehm has quit IRC (Ping timeout: 260 seconds) [20:32] *** BlueMaxim has joined #archiveteam [20:33] *** maelstrom has joined #archiveteam [20:45] Everyone: [20:45] #vinewhine for flickr and #paranormio for panoramio [20:45] both are large projects [20:45] uh [20:45] #vinewhine for vine [20:46] #flickrfckr for flickr [20:47] vine scripts will be up soon, I haven't had a very good look yet at discovery [20:47] to make sure you favorite videos are save, submit them to https://docs.google.com/forms/d/1otnjcaABgkxeao9GZKuGcjQlI_SUfHQMNRAyaKw-lZI/ [20:47] or in case of a large list, hand us a list of URLs in #vinewhine [21:07] *** lukeman has joined #archiveteam [21:08] *** Fletcher has quit IRC (Ping timeout: 244 seconds) [21:08] *** nekomune has quit IRC (Ping timeout: 244 seconds) [21:08] *** will has quit IRC (Ping timeout: 244 seconds) [21:08] *** Kenshin has quit IRC (Ping timeout: 244 seconds) [21:08] *** closure has quit IRC (Ping timeout: 244 seconds) [21:08] *** useretail has quit IRC (Ping timeout: 244 seconds) [21:08] *** i0npulse has quit IRC (Ping timeout: 244 seconds) [21:08] *** nox_ has quit IRC (Ping timeout: 244 seconds) [21:08] *** Medowar has quit IRC (Ping timeout: 244 seconds) [21:08] *** lukeman_ has quit IRC (Ping timeout: 244 seconds) [21:08] *** Marc has quit IRC (Ping timeout: 244 seconds) [21:08] *** Nemo_bis has quit IRC (Ping timeout: 244 seconds) [21:08] *** z00nx has quit IRC (Ping timeout: 244 seconds) [21:08] *** Rye has quit IRC (Ping timeout: 244 seconds) [21:08] *** will has joined #archiveteam [21:08] *** Medowar has joined #archiveteam [21:08] *** Nemo_bis has joined #archiveteam [21:08] *** Kenshin has joined #archiveteam [21:09] *** closure has joined #archiveteam [21:09] *** Rye has joined #archiveteam [21:09] *** Marc has joined #archiveteam [21:10] *** nox_ has joined #archiveteam [21:10] *** z00nx has joined #archiveteam [21:11] *** useretail has joined #archiveteam [21:13] *** i0npulse has joined #archiveteam [21:13] *** nekomune has joined #archiveteam [21:13] arkiver: you might get some incoming requests [21:13] :p [21:14] I hope so! :) [21:15] And a few more [21:24] *** Fletcher has joined #archiveteam [21:53] *** RichardG has quit IRC (Read error: Operation timed out) [21:53] *** RichardG has joined #archiveteam [22:13] *** BartoCH has quit IRC (Ping timeout: 260 seconds) [22:14] *** BartoCH has joined #archiveteam [22:20] *** kristian_ has joined #archiveteam [22:35] *** kniffy has joined #archiveteam [22:55] *** atomotic has joined #archiveteam [23:06] *** computerf has quit IRC (Bye.) [23:12] *** computerf has joined #archiveteam [23:16] *** icedice has joined #archiveteam [23:19] *** epicfacet has joined #archiveteam [23:19] *** epicfacet has quit IRC (Client Quit) [23:20] *** epicfacet has joined #archiveteam [23:20] How are you guys? [23:21] we're fine [23:21] cool [23:21] the webchat system keeps shortening my username :/ [23:21] anyways, ahve you guys ever heard of ace of spades? like the video game? [23:21] so #archiveteam is more for announcements, #archiveteam-bs is for the rest [23:22] how about site shutdowns? [23:22] yeah, you put a site that's shutting down here [23:23] so there is a site (one of the few remaining) called build and shoot, and it's been in a stage of "maintenance" for about the last three days. [23:23] they are related to AoS in the fact that they are a forum for them, and AoS has a nasty history of not backing up [23:24] which site is it? [23:24] http://buildandshoot.com/ [23:24] 99% of the content of the front page is borken/gone too, no offical word from the admins to my knowledge [23:26] what is still there is now being archive, see top of http://dashboard.at.ninjawedding.org/ [23:26] it's done [23:27] that fast? [23:27] well not a lot was left... [23:28] true. [23:28] let's move to #archiveteam-bs with further discussion on this [23:28] should i let you know if it comes back on? [23:28] yes [23:30] *** JW_work has quit IRC (Quit: Leaving.) [23:33] *** atomotic has quit IRC (Quit: My MacBook has gone to sleep. ZZZzzz…) [23:49] *** JesseW has joined #archiveteam [23:55] *** epicfacet has quit IRC (Quit: Page closed) [23:55] *** VADemon has joined #archiveteam