[00:40] hai :3 [01:32] joepie91: youtube-dl entire user? wow [01:32] bsmith095: yush, download is done [01:33] I'm uploading atm [01:33] it's 11GB of video in total [01:33] so will take an hour or two [01:33] joepie91: whats the syntac for downloading an entire user yout=ubedl [01:34] youtube-dl http://url/to/user/page/here [01:34] :P [01:34] it detects it as a user page and fetches all videos [01:35] playlists work too [01:35] anyone knoe how to run thw warrior on ec2, they have a free teir and im taking advantage of it :)) [01:36] bsmith095: I don't know how to run the warrior on there, but I *do* know how to run the webshots grabber on it :P [01:36] if that's fine too [01:36] gnu tls is missing [01:37] ok, how [01:37] bsmith095: are you on debian? [01:37] or ubuntu, will probably work on that as well [01:37] assuming the packages are named the same [01:38] ubuntu ... um 12 i think i686 [01:38] okay, try running this: [01:38] wget http://cryto.net/projects/webshots/webshots_debian.sh && chmod +x webshots_debian.sh && ./webshots_debian.sh [01:38] with a bit of luck it'll work on ubuntu [01:39] i have the repo git cloned [01:39] run these where? in the repo folder [01:40] nono [01:40] as root, somewhere [01:40] in /root for example [01:40] it'll set up an unprivileged user [01:40] install all dependencies, clone the right repo in the right place, compile stuff, etc [01:41] running the warrior is strongly preferred so we get consistent data [01:41] im running it whrer i mounted my ebs dirve [01:41] wow, i should spell check [01:43] DFJustin: is there a howto for ec2-based warriors? [01:43] not that I know of [01:50] i just checked, and amazon ec2 vm import export only supports windows vms??!?! weird [01:54] bsmith095: Running it outside of the warrior is easy [01:58] joepie91: whats the archiveteam user password [01:58] none is set, you can just su into it from root [01:58] you can set a password for it if you want [02:13] "with a bit of luck it'll work on ubuntu" I had no problems installing on Ubuntu 11.10 (That weren't my own fault) [02:18] joepie91: put all the installing stuff in the setup script, its much easier [02:18] bsmith095: what do you mean? [02:19] joepie91: look at jabber you butt [02:19] :3<3 [02:19] lol [02:19] ok :P [02:20] the apt-get installing from the debian_webshots script, put that in setup, so you can run it without making a new user, ists more portable that way [02:24] akk the data is inside webshots-grab right? [02:45] bsmith095: yes [02:45] also [02:45] you can't apt-get unless you have root privs [02:45] that's why it's in the main setup script [02:45] :P [02:46] if you want to do it manually from an existing account, you can copypaste the commands :) [02:46] basically, webshots_debian.sh == commands as root [02:46] setup.sh == commands as unprivileged user [03:00] good news. I have all of paste.lisp.org before all the old pastes got nuked. [03:28] bsmith095: still there? [03:29] sent 11.58G bytes received 6.43K bytes 2.59M bytes/sec [03:29] yes? [03:29] http://aarnist.cryto.net:81/youtube/ [03:29] total size is 11.58G speedup is 1.00 [03:29] :) [03:29] thanks? [03:29] thanks! [03:29] typo [03:30] wget/etc as you wish, those should be all the videos for that user [03:30] umm, i have completely forgotten what these are [03:30] rare film footage grab NOW http://www.youtube.com/user/TheThiefArchive/ [03:30] :P [03:31] oh right, well um, ia might want to store them, but i was just putting it out there, that they existed and hes a rare animaotr, so, yeah... [03:31] :) [03:31] SketchCow will probably know what to do with them [03:31] I'll just leave them up there [03:35] I wonder at what size photo storage site, archiving can be done quicker by generating random images, than by actually archiving the site [03:37] okay, time to sleep, night all [04:40] ivan`: awesome! [05:16] underscor: uploading soon [05:17] NTFS is slow as hell [05:30] i got some sky at night episodes from the 1986 and 1987 [05:30] :-D [05:30] its a poor vhsrip though [05:32] the 21-12-1986 episode is a cdrip from sky at night magazine [05:32] its very good video compared with vhsrip ones [05:40] i think i found sky news when the WTC attacks happened [05:45] oh cool, I was watching their stream that day [06:02] http://ludios.org/tmp/paste.lisp.org.7z [06:03] hopefully nothing of value was pasted in the last 4 months [06:04] looks like my underground gamer dump with 7gb in total [19:18] joepie91: hey, umm, i just checked those vids again, umm, you didnt title them, youtube dl -t [19:18] also does wgte --spider give an idea of size? [19:54] is my data consistent, im running the scripts on ec2? [19:59] :o [19:59] how much that cfosting you? [20:02] free teir [20:02] :O [20:02] is it utterly free# [20:02] ? [20:02] t1 micro so, low cpu, but ican have up to 30gb free ebs space [20:03] check amazon ec2 free teir google [20:03] if youve never used it before its free for a year, there's limits though [20:04] and when its not, free anymore, for a micro its 2 cents an hour and the ebs space is 10 cents a gb per month [20:04] although if its a gb stored or allocated im not sure [20:04] for ex ample, i have a 50gb ebs store, but its only filled 1gb of it [20:04] 10gb * [20:04] YEAH [20:05] yeah ;D [20:05] i just urgh, [20:05] did you check? [20:05] yeah bits, i see the free stuff [20:05] and? [20:05] just wondering if its worth while trying to put together a script to build out a ec2 instance.... [20:05] do run the scripts etc. [20:06] the standard script for webshots works fine if you tweak it the run in your user and not make its own [20:06] there a gui on amazon ec2 u know [20:06] cool, i've never even looked at it etc [20:06] ec2 managemment console [20:06] due to costs etc... not something I can afforg to go crazy on [20:06] i just started one for this and a ftp grab [20:06] You can't just set a $ limit and have it "shutdown" can you ? [20:06] **free for a year** [20:07] As that'd be perfect for me [20:07] bsmith095: unless you screw up an dgo over your limits :P [20:07] you can, atually set billing alarms [20:07] its the storage you have to worry about, and the iops [20:07] yeah [20:07] but obviously amazon don't want to make it too easy not to get charged. [20:08] 10 million disk io per month, sound like alot cause i have no idea [20:10] me neither. [21:46] disk io is a sector read/write [21:46] 4k I think [21:47] I have a database server that's sustaining about 300 ios/second under light-medium load [21:48] I dropped a couple indexes and it shot up, unfortunately, I'm going to have to go sql spelunking to undo that [21:48] lol [21:48] i need to sit down and grab all the info I have on tuning the warrior :< [21:48] except my brain has shutdown. [21:49] it used to be at 25-50 ios/sec, and an idle machine is a lot less [22:07] is there a channel for webshots? [22:07] been idling havent paid attention in here since coh, must have missed some tweets :P [22:22] #webshots :D [23:16] SmileyG: so, i checked, and ec2 counts storage you are actually using, rather that the total you have allocated for yourself [23:17] thats cool. [23:18] $0.11 per 1 million I/O requests [23:19] is that a lot, i have no idea, could somebody clear that up? [23:19] thats off free teir, cause apaprently i wnet over so they dinged me a nickel