| Time |
Nickname |
Message |
|
00:40
🔗
|
asdflol |
hai :3 |
|
01:32
🔗
|
bsmith095 |
joepie91: youtube-dl entire user? wow |
|
01:32
🔗
|
joepie91 |
bsmith095: yush, download is done |
|
01:33
🔗
|
joepie91 |
I'm uploading atm |
|
01:33
🔗
|
joepie91 |
it's 11GB of video in total |
|
01:33
🔗
|
joepie91 |
so will take an hour or two |
|
01:33
🔗
|
bsmith095 |
joepie91: whats the syntac for downloading an entire user yout=ubedl |
|
01:34
🔗
|
joepie91 |
youtube-dl http://url/to/user/page/here |
|
01:34
🔗
|
joepie91 |
:P |
|
01:34
🔗
|
joepie91 |
it detects it as a user page and fetches all videos |
|
01:35
🔗
|
DFJustin |
playlists work too |
|
01:35
🔗
|
bsmith095 |
anyone knoe how to run thw warrior on ec2, they have a free teir and im taking advantage of it :)) |
|
01:36
🔗
|
joepie91 |
bsmith095: I don't know how to run the warrior on there, but I *do* know how to run the webshots grabber on it :P |
|
01:36
🔗
|
joepie91 |
if that's fine too |
|
01:36
🔗
|
bsmith095 |
gnu tls is missing |
|
01:37
🔗
|
bsmith095 |
ok, how |
|
01:37
🔗
|
joepie91 |
bsmith095: are you on debian? |
|
01:37
🔗
|
joepie91 |
or ubuntu, will probably work on that as well |
|
01:37
🔗
|
joepie91 |
assuming the packages are named the same |
|
01:38
🔗
|
bsmith095 |
ubuntu ... um 12 i think i686 |
|
01:38
🔗
|
joepie91 |
okay, try running this: |
|
01:38
🔗
|
joepie91 |
wget http://cryto.net/projects/webshots/webshots_debian.sh && chmod +x webshots_debian.sh && ./webshots_debian.sh |
|
01:38
🔗
|
joepie91 |
with a bit of luck it'll work on ubuntu |
|
01:39
🔗
|
bsmith095 |
i have the repo git cloned |
|
01:39
🔗
|
bsmith095 |
run these where? in the repo folder |
|
01:40
🔗
|
joepie91 |
nono |
|
01:40
🔗
|
joepie91 |
as root, somewhere |
|
01:40
🔗
|
joepie91 |
in /root for example |
|
01:40
🔗
|
joepie91 |
it'll set up an unprivileged user |
|
01:40
🔗
|
joepie91 |
install all dependencies, clone the right repo in the right place, compile stuff, etc |
|
01:41
🔗
|
DFJustin |
running the warrior is strongly preferred so we get consistent data |
|
01:41
🔗
|
bsmith095 |
im running it whrer i mounted my ebs dirve |
|
01:41
🔗
|
bsmith095 |
wow, i should spell check |
|
01:43
🔗
|
bsmith095 |
DFJustin: is there a howto for ec2-based warriors? |
|
01:43
🔗
|
DFJustin |
not that I know of |
|
01:50
🔗
|
bsmith095 |
i just checked, and amazon ec2 vm import export only supports windows vms??!?! weird |
|
01:54
🔗
|
underscor |
bsmith095: Running it outside of the warrior is easy |
|
01:58
🔗
|
bsmith095 |
joepie91: whats the archiveteam user password |
|
01:58
🔗
|
joepie91 |
none is set, you can just su into it from root |
|
01:58
🔗
|
joepie91 |
you can set a password for it if you want |
|
02:13
🔗
|
Cameron_D |
"with a bit of luck it'll work on ubuntu" I had no problems installing on Ubuntu 11.10 (That weren't my own fault) |
|
02:18
🔗
|
bsmith095 |
joepie91: put all the installing stuff in the setup script, its much easier |
|
02:18
🔗
|
joepie91 |
bsmith095: what do you mean? |
|
02:19
🔗
|
asdflol |
joepie91: look at jabber you butt |
|
02:19
🔗
|
asdflol |
:3<3 |
|
02:19
🔗
|
joepie91 |
lol |
|
02:19
🔗
|
joepie91 |
ok :P |
|
02:20
🔗
|
bsmith095 |
the apt-get installing from the debian_webshots script, put that in setup, so you can run it without making a new user, ists more portable that way |
|
02:24
🔗
|
bsmith095 |
akk the data is inside webshots-grab right? |
|
02:45
🔗
|
joepie91 |
bsmith095: yes |
|
02:45
🔗
|
joepie91 |
also |
|
02:45
🔗
|
joepie91 |
you can't apt-get unless you have root privs |
|
02:45
🔗
|
joepie91 |
that's why it's in the main setup script |
|
02:45
🔗
|
joepie91 |
:P |
|
02:46
🔗
|
joepie91 |
if you want to do it manually from an existing account, you can copypaste the commands :) |
|
02:46
🔗
|
joepie91 |
basically, webshots_debian.sh == commands as root |
|
02:46
🔗
|
joepie91 |
setup.sh == commands as unprivileged user |
|
03:00
🔗
|
ivan` |
good news. I have all of paste.lisp.org before all the old pastes got nuked. |
|
03:28
🔗
|
joepie91 |
bsmith095: still there? |
|
03:29
🔗
|
joepie91 |
sent 11.58G bytes received 6.43K bytes 2.59M bytes/sec |
|
03:29
🔗
|
bsmith095 |
yes? |
|
03:29
🔗
|
joepie91 |
http://aarnist.cryto.net:81/youtube/ |
|
03:29
🔗
|
joepie91 |
total size is 11.58G speedup is 1.00 |
|
03:29
🔗
|
joepie91 |
:) |
|
03:29
🔗
|
bsmith095 |
thanks? |
|
03:29
🔗
|
bsmith095 |
thanks! |
|
03:29
🔗
|
bsmith095 |
typo |
|
03:30
🔗
|
joepie91 |
wget/etc as you wish, those should be all the videos for that user |
|
03:30
🔗
|
bsmith095 |
umm, i have completely forgotten what these are |
|
03:30
🔗
|
joepie91 |
<bsmith095>rare film footage grab NOW http://www.youtube.com/user/TheThiefArchive/ |
|
03:30
🔗
|
joepie91 |
:P |
|
03:31
🔗
|
bsmith095 |
oh right, well um, ia might want to store them, but i was just putting it out there, that they existed and hes a rare animaotr, so, yeah... |
|
03:31
🔗
|
joepie91 |
:) |
|
03:31
🔗
|
joepie91 |
SketchCow will probably know what to do with them |
|
03:31
🔗
|
joepie91 |
I'll just leave them up there |
|
03:35
🔗
|
joepie91 |
I wonder at what size photo storage site, archiving can be done quicker by generating random images, than by actually archiving the site |
|
03:37
🔗
|
joepie91 |
okay, time to sleep, night all |
|
04:40
🔗
|
underscor |
ivan`: awesome! |
|
05:16
🔗
|
ivan` |
underscor: uploading soon |
|
05:17
🔗
|
ivan` |
NTFS is slow as hell |
|
05:30
🔗
|
godane |
i got some sky at night episodes from the 1986 and 1987 |
|
05:30
🔗
|
godane |
:-D |
|
05:30
🔗
|
godane |
its a poor vhsrip though |
|
05:32
🔗
|
godane |
the 21-12-1986 episode is a cdrip from sky at night magazine |
|
05:32
🔗
|
godane |
its very good video compared with vhsrip ones |
|
05:40
🔗
|
godane |
i think i found sky news when the WTC attacks happened |
|
05:45
🔗
|
DFJustin |
oh cool, I was watching their stream that day |
|
06:02
🔗
|
ivan` |
http://ludios.org/tmp/paste.lisp.org.7z |
|
06:03
🔗
|
ivan` |
hopefully nothing of value was pasted in the last 4 months |
|
06:04
🔗
|
godane |
looks like my underground gamer dump with 7gb in total |
|
19:18
🔗
|
bsmith095 |
joepie91: hey, umm, i just checked those vids again, umm, you didnt title them, youtube dl -t |
|
19:18
🔗
|
bsmith095 |
also does wgte --spider give an idea of size? |
|
19:54
🔗
|
bsmith095 |
is my data consistent, im running the scripts on ec2? |
|
19:59
🔗
|
SmileyG |
:o |
|
19:59
🔗
|
SmileyG |
how much that cfosting you? |
|
20:02
🔗
|
bsmith095 |
free teir |
|
20:02
🔗
|
SmileyG |
:O |
|
20:02
🔗
|
SmileyG |
is it utterly free# |
|
20:02
🔗
|
SmileyG |
? |
|
20:02
🔗
|
bsmith095 |
t1 micro so, low cpu, but ican have up to 30gb free ebs space |
|
20:03
🔗
|
bsmith095 |
check amazon ec2 free teir google |
|
20:03
🔗
|
bsmith095 |
if youve never used it before its free for a year, there's limits though |
|
20:04
🔗
|
bsmith095 |
and when its not, free anymore, for a micro its 2 cents an hour and the ebs space is 10 cents a gb per month |
|
20:04
🔗
|
bsmith095 |
although if its a gb stored or allocated im not sure |
|
20:04
🔗
|
bsmith095 |
for ex ample, i have a 50gb ebs store, but its only filled 1gb of it |
|
20:04
🔗
|
bsmith095 |
10gb * |
|
20:04
🔗
|
SmileyG |
YEAH |
|
20:05
🔗
|
SmileyG |
yeah ;D |
|
20:05
🔗
|
SmileyG |
i just urgh, |
|
20:05
🔗
|
bsmith095 |
did you check? |
|
20:05
🔗
|
SmileyG |
yeah bits, i see the free stuff |
|
20:05
🔗
|
bsmith095 |
and? |
|
20:05
🔗
|
SmileyG |
just wondering if its worth while trying to put together a script to build out a ec2 instance.... |
|
20:05
🔗
|
SmileyG |
do run the scripts etc. |
|
20:06
🔗
|
bsmith095 |
the standard script for webshots works fine if you tweak it the run in your user and not make its own |
|
20:06
🔗
|
bsmith095 |
there a gui on amazon ec2 u know |
|
20:06
🔗
|
SmileyG |
cool, i've never even looked at it etc |
|
20:06
🔗
|
bsmith095 |
ec2 managemment console |
|
20:06
🔗
|
SmileyG |
due to costs etc... not something I can afforg to go crazy on |
|
20:06
🔗
|
bsmith095 |
i just started one for this and a ftp grab |
|
20:06
🔗
|
SmileyG |
You can't just set a $ limit and have it "shutdown" can you ? |
|
20:06
🔗
|
bsmith095 |
**free for a year** |
|
20:07
🔗
|
SmileyG |
As that'd be perfect for me |
|
20:07
🔗
|
SmileyG |
bsmith095: unless you screw up an dgo over your limits :P |
|
20:07
🔗
|
bsmith095 |
you can, atually set billing alarms |
|
20:07
🔗
|
bsmith095 |
its the storage you have to worry about, and the iops |
|
20:07
🔗
|
SmileyG |
yeah |
|
20:07
🔗
|
SmileyG |
but obviously amazon don't want to make it too easy not to get charged. |
|
20:08
🔗
|
bsmith095 |
10 million disk io per month, sound like alot cause i have no idea |
|
20:10
🔗
|
SmileyG |
me neither. |
|
21:46
🔗
|
chronomex |
disk io is a sector read/write |
|
21:46
🔗
|
chronomex |
4k I think |
|
21:47
🔗
|
chronomex |
I have a database server that's sustaining about 300 ios/second under light-medium load |
|
21:48
🔗
|
chronomex |
I dropped a couple indexes and it shot up, unfortunately, I'm going to have to go sql spelunking to undo that |
|
21:48
🔗
|
SmileyG |
lol |
|
21:48
🔗
|
SmileyG |
i need to sit down and grab all the info I have on tuning the warrior :< |
|
21:48
🔗
|
SmileyG |
except my brain has shutdown. |
|
21:49
🔗
|
chronomex |
it used to be at 25-50 ios/sec, and an idle machine is a lot less |
|
22:07
🔗
|
S[h]O[r]T |
is there a channel for webshots? |
|
22:07
🔗
|
S[h]O[r]T |
been idling havent paid attention in here since coh, must have missed some tweets :P |
|
22:22
🔗
|
SmileyG |
#webshots :D |
|
23:16
🔗
|
bsmith095 |
SmileyG: so, i checked, and ec2 counts storage you are actually using, rather that the total you have allocated for yourself |
|
23:17
🔗
|
SmileyG |
thats cool. |
|
23:18
🔗
|
bsmith095 |
$0.11 per 1 million I/O requests |
|
23:19
🔗
|
bsmith095 |
is that a lot, i have no idea, could somebody clear that up? |
|
23:19
🔗
|
bsmith095 |
thats off free teir, cause apaprently i wnet over so they dinged me a nickel |