#archiveteam 2016-05-08,Sun

↑back Search

Time Nickname Message
00:02 🔗 BlueMaxim has joined #archiveteam
00:25 🔗 icedice has quit IRC (Quit: Leaving)
00:30 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
00:30 🔗 nwf has joined #archiveteam
00:37 🔗 JesseW has joined #archiveteam
00:47 🔗 TC01 has quit IRC (Read error: Operation timed out)
00:57 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
01:14 🔗 Microguru has joined #archiveteam
01:15 🔗 nwf has joined #archiveteam
01:19 🔗 BartoCH has quit IRC (Ping timeout: 260 seconds)
01:23 🔗 BlueMaxim has quit IRC (Quit: Leaving)
01:23 🔗 BartoCH has joined #archiveteam
01:28 🔗 JesseW has quit IRC (Ping timeout: 370 seconds)
01:49 🔗 BnA-Rob1n has quit IRC (Read error: Connection reset by peer)
01:49 🔗 Kazzy has quit IRC (hub.se efnet.portlane.se)
01:49 🔗 Igloo has quit IRC (hub.se efnet.portlane.se)
01:49 🔗 Fletcher_ has quit IRC (hub.se efnet.portlane.se)
01:49 🔗 koon has quit IRC (hub.se efnet.portlane.se)
01:49 🔗 superkuh has joined #archiveteam
01:50 🔗 Sk1d has quit IRC (Ping timeout: 194 seconds)
01:51 🔗 Nemo_bis has quit IRC (Ping timeout: 244 seconds)
02:06 🔗 TC01 has joined #archiveteam
02:11 🔗 BnA-Rob1n has joined #archiveteam
02:11 🔗 Nemo_bis has joined #archiveteam
02:14 🔗 superkuh has quit IRC (hub.se efnet.port80.se)
02:14 🔗 BartoCH has quit IRC (hub.se efnet.port80.se)
02:14 🔗 hive-mind has quit IRC (hub.se efnet.port80.se)
02:14 🔗 zhongfu has quit IRC (hub.se efnet.port80.se)
02:14 🔗 philpem has quit IRC (hub.se efnet.port80.se)
02:14 🔗 Start has quit IRC (hub.se efnet.port80.se)
02:14 🔗 r3c0d3x has quit IRC (hub.se efnet.port80.se)
02:14 🔗 lesderid has quit IRC (hub.se efnet.port80.se)
02:14 🔗 sigkell has quit IRC (hub.se efnet.port80.se)
02:14 🔗 pfallenop has quit IRC (hub.se efnet.port80.se)
02:14 🔗 Meroje has quit IRC (hub.se efnet.port80.se)
02:14 🔗 Coderjoe has quit IRC (hub.se efnet.port80.se)
02:14 🔗 bauruine has quit IRC (hub.se efnet.port80.se)
02:14 🔗 Jon has quit IRC (hub.se efnet.port80.se)
02:14 🔗 FalconK has quit IRC (hub.se efnet.port80.se)
02:14 🔗 balrog has quit IRC (hub.se efnet.port80.se)
02:14 🔗 d_rebel has quit IRC (hub.se efnet.port80.se)
02:14 🔗 zerkalo has quit IRC (hub.se efnet.port80.se)
02:14 🔗 SirCmpwn has quit IRC (hub.se efnet.port80.se)
02:14 🔗 szalwia has quit IRC (hub.se efnet.port80.se)
02:14 🔗 antonizoo has quit IRC (hub.se efnet.port80.se)
02:14 🔗 Atluxity has quit IRC (hub.se efnet.port80.se)
02:14 🔗 Sanqui has quit IRC (hub.se efnet.port80.se)
02:14 🔗 hictooth has quit IRC (hub.se efnet.port80.se)
02:14 🔗 davidar has quit IRC (hub.se efnet.port80.se)
02:14 🔗 Rickster has quit IRC (hub.se efnet.port80.se)
02:14 🔗 Muad-Dib has quit IRC (hub.se efnet.port80.se)
02:14 🔗 Famicoman has quit IRC (hub.se efnet.port80.se)
02:14 🔗 _desu___ has quit IRC (hub.se efnet.port80.se)
02:14 🔗 pikhq has quit IRC (hub.se efnet.port80.se)
02:14 🔗 bai has quit IRC (hub.se efnet.port80.se)
02:14 🔗 dan- has quit IRC (hub.se efnet.port80.se)
02:14 🔗 kevin has quit IRC (hub.se efnet.port80.se)
02:14 🔗 karissa__ has quit IRC (hub.se efnet.port80.se)
02:14 🔗 JSharp___ has quit IRC (hub.se efnet.port80.se)
02:14 🔗 victor has quit IRC (hub.se efnet.port80.se)
02:14 🔗 VonGuard has quit IRC (hub.se efnet.port80.se)
02:14 🔗 Vito` has quit IRC (hub.se efnet.port80.se)
02:14 🔗 Boltsie has quit IRC (hub.se efnet.port80.se)
02:14 🔗 TheKiwi has quit IRC (hub.se efnet.port80.se)
02:14 🔗 HCross2 has quit IRC (hub.se efnet.port80.se)
02:14 🔗 Ctrl-S___ has quit IRC (hub.se efnet.port80.se)
02:14 🔗 deathy has quit IRC (hub.se efnet.port80.se)
02:14 🔗 johtso has quit IRC (hub.se efnet.port80.se)
02:53 🔗 WinterFox has joined #archiveteam
03:01 🔗 Ravenloft has joined #archiveteam
03:06 🔗 balrog has joined #archiveteam
03:06 🔗 swebb sets mode: +o balrog
03:07 🔗 JesseW has joined #archiveteam
03:16 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
03:20 🔗 nwf has joined #archiveteam
03:40 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
03:40 🔗 nwf has joined #archiveteam
03:41 🔗 JesseW has quit IRC (Ping timeout: 370 seconds)
03:50 🔗 JesseW has joined #archiveteam
04:19 🔗 vitzli has joined #archiveteam
04:22 🔗 BlueMaxim has joined #archiveteam
04:41 🔗 zhongfu has joined #archiveteam
04:41 🔗 superkuh has joined #archiveteam
04:41 🔗 BartoCH has joined #archiveteam
04:41 🔗 hive-mind has joined #archiveteam
04:41 🔗 philpem has joined #archiveteam
04:41 🔗 Start has joined #archiveteam
04:41 🔗 r3c0d3x has joined #archiveteam
04:41 🔗 lesderid has joined #archiveteam
04:41 🔗 sigkell has joined #archiveteam
04:41 🔗 pfallenop has joined #archiveteam
04:41 🔗 Meroje has joined #archiveteam
04:41 🔗 Coderjoe has joined #archiveteam
04:41 🔗 bauruine has joined #archiveteam
04:41 🔗 Jon has joined #archiveteam
04:41 🔗 FalconK has joined #archiveteam
04:41 🔗 d_rebel has joined #archiveteam
04:41 🔗 zerkalo has joined #archiveteam
04:41 🔗 SirCmpwn has joined #archiveteam
04:41 🔗 szalwia has joined #archiveteam
04:41 🔗 antonizoo has joined #archiveteam
04:41 🔗 Atluxity has joined #archiveteam
04:41 🔗 Sanqui has joined #archiveteam
04:41 🔗 hictooth has joined #archiveteam
04:41 🔗 davidar has joined #archiveteam
04:41 🔗 Rickster has joined #archiveteam
04:41 🔗 Muad-Dib has joined #archiveteam
04:41 🔗 Famicoman has joined #archiveteam
04:41 🔗 _desu___ has joined #archiveteam
04:41 🔗 pikhq has joined #archiveteam
04:41 🔗 bai has joined #archiveteam
04:41 🔗 dan- has joined #archiveteam
04:41 🔗 kevin has joined #archiveteam
04:41 🔗 karissa__ has joined #archiveteam
04:41 🔗 JSharp___ has joined #archiveteam
04:41 🔗 victor has joined #archiveteam
04:41 🔗 Ctrl-S___ has joined #archiveteam
04:41 🔗 VonGuard has joined #archiveteam
04:41 🔗 Vito` has joined #archiveteam
04:41 🔗 HCross2 has joined #archiveteam
04:41 🔗 Boltsie has joined #archiveteam
04:41 🔗 TheKiwi has joined #archiveteam
04:41 🔗 deathy has joined #archiveteam
04:41 🔗 johtso has joined #archiveteam
04:41 🔗 efnet.port80.se sets mode: +oo Atluxity HCross2
04:41 🔗 swebb sets mode: +o Atluxity
04:42 🔗 Kazzy has joined #archiveteam
04:42 🔗 Igloo has joined #archiveteam
04:42 🔗 Fletcher_ has joined #archiveteam
04:42 🔗 koon has joined #archiveteam
04:43 🔗 Sk1d has joined #archiveteam
05:07 🔗 ralphdnak has joined #archiveteam
05:18 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
05:21 🔗 nwf has joined #archiveteam
05:27 🔗 dashcloud has quit IRC (Read error: Operation timed out)
05:31 🔗 dashcloud has joined #archiveteam
05:34 🔗 edward81 has joined #archiveteam
05:40 🔗 Microguru has quit IRC (Quit: Microguru)
05:52 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
05:58 🔗 metalcamp has joined #archiveteam
05:59 🔗 nwf has joined #archiveteam
06:01 🔗 VADemon has quit IRC (Quit: left4dead)
06:14 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
06:15 🔗 nwf has joined #archiveteam
06:25 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
06:32 🔗 JesseW I don't see arto in the warrior tracker. arkiver -- how are you running it?
06:41 🔗 nwf has joined #archiveteam
06:45 🔗 sHATNER has joined #archiveteam
07:14 🔗 JesseW has quit IRC (Ping timeout: 370 seconds)
07:14 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
07:23 🔗 nwf has joined #archiveteam
07:49 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
07:50 🔗 nwf has joined #archiveteam
08:42 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
08:43 🔗 nwf has joined #archiveteam
08:54 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
08:54 🔗 pfallenop has quit IRC (Ping timeout: 260 seconds)
09:06 🔗 r3c0d3x has quit IRC (Ping timeout: 260 seconds)
09:06 🔗 r3c0d3x has joined #archiveteam
09:31 🔗 nwf has joined #archiveteam
09:56 🔗 Ravenloft has quit IRC (Read error: Connection reset by peer)
09:56 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
09:58 🔗 nwf has joined #archiveteam
10:09 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
10:10 🔗 nwf has joined #archiveteam
10:13 🔗 ralphdnak has quit IRC (Read error: Operation timed out)
10:20 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
10:21 🔗 nwf has joined #archiveteam
10:33 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
10:38 🔗 nwf has joined #archiveteam
10:54 🔗 pfallenop has joined #archiveteam
11:00 🔗 luckcolor has quit IRC (http://quassel-irc.org - Chat comfortably. Anywhere.)
11:00 🔗 luckcolor has joined #archiveteam
11:02 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
11:02 🔗 luckcolor has quit IRC (Client Quit)
11:02 🔗 luckcolor has joined #archiveteam
11:03 🔗 luckcolor has quit IRC (Client Quit)
11:03 🔗 luckcolor has joined #archiveteam
11:05 🔗 luckcolor has quit IRC (Client Quit)
11:07 🔗 luckcolor has joined #archiveteam
11:07 🔗 luckcolor has quit IRC (Client Quit)
11:07 🔗 luckcolor has joined #archiveteam
11:11 🔗 nwf has joined #archiveteam
11:16 🔗 BlueMaxim has quit IRC (Quit: Leaving)
11:42 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
11:42 🔗 nwf has joined #archiveteam
11:49 🔗 arkiver JesseW: Scripts are ready, I didn't start it yet
11:49 🔗 arkiver I try to never start project only an hour before I go to bed
11:49 🔗 arkiver Often we need to make limiting changes and some small script fixes/additions
11:52 🔗 luckcolor https://github.com/bevacqua/shots
11:52 🔗 luckcolor Animated gif of archive.org website history
11:54 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
12:01 🔗 nwf has joined #archiveteam
12:21 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
12:21 🔗 nwf has joined #archiveteam
13:32 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
13:43 🔗 nwf has joined #archiveteam
13:49 🔗 WinterFox has quit IRC (Remote host closed the connection)
14:10 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
14:12 🔗 nwf has joined #archiveteam
14:19 🔗 atomotic has joined #archiveteam
14:22 🔗 SN4T14 has quit IRC (Remote host closed the connection)
14:48 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
14:49 🔗 Wuked has joined #archiveteam
15:28 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
15:32 🔗 nwf has joined #archiveteam
16:08 🔗 atomotic has joined #archiveteam
16:37 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
16:51 🔗 nwf has joined #archiveteam
17:16 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
17:16 🔗 nwf has joined #archiveteam
17:29 🔗 JesseW has joined #archiveteam
17:29 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
17:42 🔗 vitzli has quit IRC (Quit: Leaving)
17:56 🔗 pfallenop has quit IRC (Ping timeout: 260 seconds)
17:58 🔗 xXx_ndidd has joined #archiveteam
18:03 🔗 pfallenop has joined #archiveteam
18:08 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
18:09 🔗 nwf has joined #archiveteam
18:09 🔗 bsmith093 has quit IRC (Ping timeout: 370 seconds)
18:10 🔗 ndiddy has quit IRC (Read error: Operation timed out)
18:15 🔗 bsmith093 has joined #archiveteam
18:18 🔗 xXx_ndidd is now known as ndiddy
18:29 🔗 pfallenop has quit IRC (Ping timeout: 260 seconds)
18:32 🔗 logchfoo2 starts logging #archiveteam at Sun May 08 18:32:19 2016
18:32 🔗 logchfoo2 has joined #archiveteam
18:33 🔗 Fake-Nam1 has joined #archiveteam
18:33 🔗 schbirid has quit IRC (hub.efnet.us irc.Prison.NET)
18:33 🔗 RichardG has quit IRC (hub.efnet.us irc.Prison.NET)
18:33 🔗 Fake-Name has quit IRC (hub.efnet.us irc.Prison.NET)
18:33 🔗 vOYtEC has quit IRC (hub.efnet.us irc.Prison.NET)
18:33 🔗 logchfoo1 has quit IRC (hub.efnet.us irc.Prison.NET)
18:33 🔗 Zebranky has quit IRC (hub.efnet.us irc.Prison.NET)
18:33 🔗 Peetz0r has quit IRC (hub.efnet.us irc.Prison.NET)
18:33 🔗 Infreq has quit IRC (hub.efnet.us irc.Prison.NET)
18:33 🔗 sivoais has quit IRC (hub.efnet.us irc.Prison.NET)
18:33 🔗 achip has quit IRC (hub.efnet.us irc.Prison.NET)
18:35 🔗 RichardG_ has joined #archiveteam
18:37 🔗 Zebranky_ has joined #archiveteam
18:39 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
18:40 🔗 nwf has joined #archiveteam
18:43 🔗 sivoais_ has joined #archiveteam
18:48 🔗 schbirid2 has joined #archiveteam
18:56 🔗 Nemo_bis SketchCow: did you ever publish the script you use to add subjects to items? (I have someone who'd like to add subjects to some Italian-language books.) I checked https://github.com/internetarchive/collections-cleaners
19:09 🔗 pfallenop has joined #archiveteam
19:21 🔗 Peetz0r has joined #archiveteam
19:21 🔗 Wuked has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…)
19:21 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
19:33 🔗 nwf has joined #archiveteam
19:53 🔗 BartoCH has quit IRC (Ping timeout: 260 seconds)
20:00 🔗 BartoCH has joined #archiveteam
20:12 🔗 schbirid2 has quit IRC (Quit: Leaving)
20:23 🔗 Ravenloft has joined #archiveteam
20:26 🔗 edward81 has quit IRC (Ping timeout: 492 seconds)
20:31 🔗 edward81 has joined #archiveteam
20:32 🔗 arkiver For arto.com:
20:32 🔗 ariscop has quit IRC (Read error: Operation timed out)
20:33 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
20:34 🔗 Ravenloft has quit IRC (Ping timeout: 633 seconds)
20:34 🔗 nwf has joined #archiveteam
20:42 🔗 arkiver Videos on arto.com are streamed using rtmp
20:43 🔗 arkiver which we can't download with wget
20:43 🔗 JesseW has quit IRC (Ping timeout: 370 seconds)
20:43 🔗 arkiver There are a low resolution and a high resolution videos for each video
20:44 🔗 arkiver These rtmp streams are used by the flash viewer
20:44 🔗 arkiver arto.com also has some support for html5 in the source code
20:44 🔗 arkiver The html does have the high-res video as a normal http URL, so we can download that
20:45 🔗 edward81 has quit IRC (Read error: Operation timed out)
20:45 🔗 arkiver I haven't seen a good option though in arto.com to switch to html5, so videos might not play in the Wayback Machine
20:47 🔗 Honno_ has quit IRC (Read error: Operation timed out)
20:47 🔗 HCross could we get the rtmp links out then run a script only project to get them?
20:48 🔗 arkiver The hd rtmp version is the exact same as the http video from the html5 support
20:48 🔗 arkiver 'provider': 'rtmp',
20:48 🔗 arkiver 'streamer': 'rtmp://artovideos.cloud2.artodata.com/cfx/st/',
20:48 🔗 arkiver 'file': 'data/user/video/videos/284/2841f3d3-5176-4cf1-8028-11ed777d3cc2.mp4',
20:48 🔗 arkiver 'plugins': 'hd-1',
20:48 🔗 arkiver 'hd.file': 'data/user/video/videos/0d0/0d066d26-c4b8-4da1-9545-ff5fd51bd26e.mp4',
20:48 🔗 arkiver see hd.file
20:48 🔗 arkiver html5:
20:48 🔗 arkiver { type: 'html5', config: { 'file': 'http://artovideos.cloud2.artodata.com.s3.amazonaws.com/data/user/video/videos/0d0/0d066d26-c4b8-4da1-9545-ff5fd51bd26e.mp4', 'provider': 'video' } }
20:49 🔗 midas we can try to grab the s3 bucket
20:49 🔗 arkiver Might be a good thing to do
20:49 🔗 arkiver I'm first starting the project though
20:49 🔗 arkiver Getting lists of URLs to ignore in now
20:50 🔗 midas ill have a look at the s3 bucket in the morning
20:50 🔗 arkiver Would be great!
21:16 🔗 ariscop has joined #archiveteam
21:24 🔗 arkiver arto-grab script are online
21:30 🔗 Wuked has joined #archiveteam
21:31 🔗 Wuked has quit IRC (Client Quit)
21:36 🔗 arkiver items for arto are loaded
21:36 🔗 arkiver arto is started!
21:36 🔗 arkiver let me know if you see anything strange
21:37 🔗 Medowar FIRST!
21:37 🔗 arkiver we're going through 6 million IDs
21:37 🔗 Medowar holy smokes, arto is fast. What service are they using?
21:38 🔗 Medowar aws?
21:39 🔗 metalcamp has quit IRC (Ping timeout: 244 seconds)
21:41 🔗 Medowar looks good until now, a few 403s, that get retried a few times and then ignored
21:42 🔗 arkiver do you have a log for me?
21:42 🔗 arkiver I'd like to check if it's going ok
21:43 🔗 Kazzy do we have a channel for arto?
21:43 🔗 arkiver I don't think so
21:44 🔗 Medowar http://pastebin.com/QyFUWb08
21:45 🔗 Medowar aaaaand rate limited
21:45 🔗 Medowar want a full log?
21:46 🔗 arkiver Nah, looks fine
21:46 🔗 Kazzy Medowar: artodata.com seems to be coming from aws, yeah
21:47 🔗 Medowar yeah saw that, seems crazy fast
21:47 🔗 Medowar firefox starts to lag, when 6 threats are running
21:47 🔗 SN4T14 has joined #archiveteam
21:51 🔗 Kazzy i'm seeing ~25% cpu usage with 20 threads
21:52 🔗 arkiver I'm not sure if they ban and what happends if you're banned, so let me know if you see anything
21:54 🔗 Kazzy i can *try* to get myself banned if you're interested in finding out..
21:54 🔗 arkiver nah, no need for that
21:54 🔗 Kazzy chances are with autoscaling they'll start banning at some point
21:55 🔗 arkiver we'll see what happens
21:57 🔗 HCross2 We'll just cost them a lot
21:57 🔗 Medowar since they are only using cloudfront/S3 and not ec2 or elb etc, they might not even have a banning system in place
21:57 🔗 arkiver HCross2: They could have expected this :)
21:58 🔗 HCross2 I think S3 rate limits
21:59 🔗 Medowar they are not exposing s3, only to cloudfront, so rate limit might not trigger
22:02 🔗 JesseW has joined #archiveteam
22:15 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
22:16 🔗 nwf has joined #archiveteam
22:20 🔗 Medowar ok, my cluster is up, checking load tomorrow and then seeing if I can fire up more workers
22:41 🔗 HCross2 Do we need a lot of workers?
22:41 🔗 arkiver I'm not sure
22:41 🔗 arkiver You can fire some up, see how it goes
22:47 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
22:48 🔗 nwf has joined #archiveteam
23:12 🔗 BlueMaxim has joined #archiveteam
23:20 🔗 Ravenloft has joined #archiveteam
23:28 🔗 RichardG_ is now known as RichardG
23:29 🔗 nwf has quit IRC (Ping timeout: 633 seconds)
23:29 🔗 nwf has joined #archiveteam
23:41 🔗 SN4T14 has quit IRC (Remote host closed the connection)
23:48 🔗 RichardG has quit IRC (Quit: Keyboard not found, press F1 to continue)

irclogger-viewer