[00:02] *** BlueMaxim has joined #archiveteam [00:25] *** icedice has quit IRC (Quit: Leaving) [00:30] *** nwf has quit IRC (Ping timeout: 633 seconds) [00:30] *** nwf has joined #archiveteam [00:37] *** JesseW has joined #archiveteam [00:47] *** TC01 has quit IRC (Read error: Operation timed out) [00:57] *** nwf has quit IRC (Ping timeout: 633 seconds) [01:14] *** Microguru has joined #archiveteam [01:15] *** nwf has joined #archiveteam [01:19] *** BartoCH has quit IRC (Ping timeout: 260 seconds) [01:23] *** BlueMaxim has quit IRC (Quit: Leaving) [01:23] *** BartoCH has joined #archiveteam [01:28] *** JesseW has quit IRC (Ping timeout: 370 seconds) [01:49] *** BnA-Rob1n has quit IRC (Read error: Connection reset by peer) [01:49] *** Kazzy has quit IRC (hub.se efnet.portlane.se) [01:49] *** Igloo has quit IRC (hub.se efnet.portlane.se) [01:49] *** Fletcher_ has quit IRC (hub.se efnet.portlane.se) [01:49] *** koon has quit IRC (hub.se efnet.portlane.se) [01:49] *** superkuh has joined #archiveteam [01:50] *** Sk1d has quit IRC (Ping timeout: 194 seconds) [01:51] *** Nemo_bis has quit IRC (Ping timeout: 244 seconds) [02:06] *** TC01 has joined #archiveteam [02:11] *** BnA-Rob1n has joined #archiveteam [02:11] *** Nemo_bis has joined #archiveteam [02:14] *** superkuh has quit IRC (hub.se efnet.port80.se) [02:14] *** BartoCH has quit IRC (hub.se efnet.port80.se) [02:14] *** hive-mind has quit IRC (hub.se efnet.port80.se) [02:14] *** zhongfu has quit IRC (hub.se efnet.port80.se) [02:14] *** philpem has quit IRC (hub.se efnet.port80.se) [02:14] *** Start has quit IRC (hub.se efnet.port80.se) [02:14] *** r3c0d3x has quit IRC (hub.se efnet.port80.se) [02:14] *** lesderid has quit IRC (hub.se efnet.port80.se) [02:14] *** sigkell has quit IRC (hub.se efnet.port80.se) [02:14] *** pfallenop has quit IRC (hub.se efnet.port80.se) [02:14] *** Meroje has quit IRC (hub.se efnet.port80.se) [02:14] *** Coderjoe has quit IRC (hub.se efnet.port80.se) [02:14] *** bauruine has quit IRC (hub.se efnet.port80.se) [02:14] *** Jon has quit IRC (hub.se efnet.port80.se) [02:14] *** FalconK has quit IRC (hub.se efnet.port80.se) [02:14] *** balrog has quit IRC (hub.se efnet.port80.se) [02:14] *** d_rebel has quit IRC (hub.se efnet.port80.se) [02:14] *** zerkalo has quit IRC (hub.se efnet.port80.se) [02:14] *** SirCmpwn has quit IRC (hub.se efnet.port80.se) [02:14] *** szalwia has quit IRC (hub.se efnet.port80.se) [02:14] *** antonizoo has quit IRC (hub.se efnet.port80.se) [02:14] *** Atluxity has quit IRC (hub.se efnet.port80.se) [02:14] *** Sanqui has quit IRC (hub.se efnet.port80.se) [02:14] *** hictooth has quit IRC (hub.se efnet.port80.se) [02:14] *** davidar has quit IRC (hub.se efnet.port80.se) [02:14] *** Rickster has quit IRC (hub.se efnet.port80.se) [02:14] *** Muad-Dib has quit IRC (hub.se efnet.port80.se) [02:14] *** Famicoman has quit IRC (hub.se efnet.port80.se) [02:14] *** _desu___ has quit IRC (hub.se efnet.port80.se) [02:14] *** pikhq has quit IRC (hub.se efnet.port80.se) [02:14] *** bai has quit IRC (hub.se efnet.port80.se) [02:14] *** dan- has quit IRC (hub.se efnet.port80.se) [02:14] *** kevin has quit IRC (hub.se efnet.port80.se) [02:14] *** karissa__ has quit IRC (hub.se efnet.port80.se) [02:14] *** JSharp___ has quit IRC (hub.se efnet.port80.se) [02:14] *** victor has quit IRC (hub.se efnet.port80.se) [02:14] *** VonGuard has quit IRC (hub.se efnet.port80.se) [02:14] *** Vito` has quit IRC (hub.se efnet.port80.se) [02:14] *** Boltsie has quit IRC (hub.se efnet.port80.se) [02:14] *** TheKiwi has quit IRC (hub.se efnet.port80.se) [02:14] *** HCross2 has quit IRC (hub.se efnet.port80.se) [02:14] *** Ctrl-S___ has quit IRC (hub.se efnet.port80.se) [02:14] *** deathy has quit IRC (hub.se efnet.port80.se) [02:14] *** johtso has quit IRC (hub.se efnet.port80.se) [02:53] *** WinterFox has joined #archiveteam [03:01] *** Ravenloft has joined #archiveteam [03:06] *** balrog has joined #archiveteam [03:06] *** swebb sets mode: +o balrog [03:07] *** JesseW has joined #archiveteam [03:16] *** nwf has quit IRC (Ping timeout: 633 seconds) [03:20] *** nwf has joined #archiveteam [03:40] *** nwf has quit IRC (Ping timeout: 633 seconds) [03:40] *** nwf has joined #archiveteam [03:41] *** JesseW has quit IRC (Ping timeout: 370 seconds) [03:50] *** JesseW has joined #archiveteam [04:19] *** vitzli has joined #archiveteam [04:22] *** BlueMaxim has joined #archiveteam [04:41] *** zhongfu has joined #archiveteam [04:41] *** superkuh has joined #archiveteam [04:41] *** BartoCH has joined #archiveteam [04:41] *** hive-mind has joined #archiveteam [04:41] *** philpem has joined #archiveteam [04:41] *** Start has joined #archiveteam [04:41] *** r3c0d3x has joined #archiveteam [04:41] *** lesderid has joined #archiveteam [04:41] *** sigkell has joined #archiveteam [04:41] *** pfallenop has joined #archiveteam [04:41] *** Meroje has joined #archiveteam [04:41] *** Coderjoe has joined #archiveteam [04:41] *** bauruine has joined #archiveteam [04:41] *** Jon has joined #archiveteam [04:41] *** FalconK has joined #archiveteam [04:41] *** d_rebel has joined #archiveteam [04:41] *** zerkalo has joined #archiveteam [04:41] *** SirCmpwn has joined #archiveteam [04:41] *** szalwia has joined #archiveteam [04:41] *** antonizoo has joined #archiveteam [04:41] *** Atluxity has joined #archiveteam [04:41] *** Sanqui has joined #archiveteam [04:41] *** hictooth has joined #archiveteam [04:41] *** davidar has joined #archiveteam [04:41] *** Rickster has joined #archiveteam [04:41] *** Muad-Dib has joined #archiveteam [04:41] *** Famicoman has joined #archiveteam [04:41] *** _desu___ has joined #archiveteam [04:41] *** pikhq has joined #archiveteam [04:41] *** bai has joined #archiveteam [04:41] *** dan- has joined #archiveteam [04:41] *** kevin has joined #archiveteam [04:41] *** karissa__ has joined #archiveteam [04:41] *** JSharp___ has joined #archiveteam [04:41] *** victor has joined #archiveteam [04:41] *** Ctrl-S___ has joined #archiveteam [04:41] *** VonGuard has joined #archiveteam [04:41] *** Vito` has joined #archiveteam [04:41] *** HCross2 has joined #archiveteam [04:41] *** Boltsie has joined #archiveteam [04:41] *** TheKiwi has joined #archiveteam [04:41] *** deathy has joined #archiveteam [04:41] *** johtso has joined #archiveteam [04:41] *** efnet.port80.se sets mode: +oo Atluxity HCross2 [04:41] *** swebb sets mode: +o Atluxity [04:42] *** Kazzy has joined #archiveteam [04:42] *** Igloo has joined #archiveteam [04:42] *** Fletcher_ has joined #archiveteam [04:42] *** koon has joined #archiveteam [04:43] *** Sk1d has joined #archiveteam [05:07] *** ralphdnak has joined #archiveteam [05:18] *** nwf has quit IRC (Ping timeout: 633 seconds) [05:21] *** nwf has joined #archiveteam [05:27] *** dashcloud has quit IRC (Read error: Operation timed out) [05:31] *** dashcloud has joined #archiveteam [05:34] *** edward81 has joined #archiveteam [05:40] *** Microguru has quit IRC (Quit: Microguru) [05:52] *** nwf has quit IRC (Ping timeout: 633 seconds) [05:58] *** metalcamp has joined #archiveteam [05:59] *** nwf has joined #archiveteam [06:01] *** VADemon has quit IRC (Quit: left4dead) [06:14] *** nwf has quit IRC (Ping timeout: 633 seconds) [06:15] *** nwf has joined #archiveteam [06:25] *** nwf has quit IRC (Ping timeout: 633 seconds) [06:32] I don't see arto in the warrior tracker. arkiver -- how are you running it? [06:41] *** nwf has joined #archiveteam [06:45] *** sHATNER has joined #archiveteam [07:14] *** JesseW has quit IRC (Ping timeout: 370 seconds) [07:14] *** nwf has quit IRC (Ping timeout: 633 seconds) [07:23] *** nwf has joined #archiveteam [07:49] *** nwf has quit IRC (Ping timeout: 633 seconds) [07:50] *** nwf has joined #archiveteam [08:42] *** nwf has quit IRC (Ping timeout: 633 seconds) [08:43] *** nwf has joined #archiveteam [08:54] *** nwf has quit IRC (Ping timeout: 633 seconds) [08:54] *** pfallenop has quit IRC (Ping timeout: 260 seconds) [09:06] *** r3c0d3x has quit IRC (Ping timeout: 260 seconds) [09:06] *** r3c0d3x has joined #archiveteam [09:31] *** nwf has joined #archiveteam [09:56] *** Ravenloft has quit IRC (Read error: Connection reset by peer) [09:56] *** nwf has quit IRC (Ping timeout: 633 seconds) [09:58] *** nwf has joined #archiveteam [10:09] *** nwf has quit IRC (Ping timeout: 633 seconds) [10:10] *** nwf has joined #archiveteam [10:13] *** ralphdnak has quit IRC (Read error: Operation timed out) [10:20] *** nwf has quit IRC (Ping timeout: 633 seconds) [10:21] *** nwf has joined #archiveteam [10:33] *** nwf has quit IRC (Ping timeout: 633 seconds) [10:38] *** nwf has joined #archiveteam [10:54] *** pfallenop has joined #archiveteam [11:00] *** luckcolor has quit IRC (http://quassel-irc.org - Chat comfortably. Anywhere.) [11:00] *** luckcolor has joined #archiveteam [11:02] *** nwf has quit IRC (Ping timeout: 633 seconds) [11:02] *** luckcolor has quit IRC (Client Quit) [11:02] *** luckcolor has joined #archiveteam [11:03] *** luckcolor has quit IRC (Client Quit) [11:03] *** luckcolor has joined #archiveteam [11:05] *** luckcolor has quit IRC (Client Quit) [11:07] *** luckcolor has joined #archiveteam [11:07] *** luckcolor has quit IRC (Client Quit) [11:07] *** luckcolor has joined #archiveteam [11:11] *** nwf has joined #archiveteam [11:16] *** BlueMaxim has quit IRC (Quit: Leaving) [11:42] *** nwf has quit IRC (Ping timeout: 633 seconds) [11:42] *** nwf has joined #archiveteam [11:49] JesseW: Scripts are ready, I didn't start it yet [11:49] I try to never start project only an hour before I go to bed [11:49] Often we need to make limiting changes and some small script fixes/additions [11:52] https://github.com/bevacqua/shots [11:52] Animated gif of archive.org website history [11:54] *** nwf has quit IRC (Ping timeout: 633 seconds) [12:01] *** nwf has joined #archiveteam [12:21] *** nwf has quit IRC (Ping timeout: 633 seconds) [12:21] *** nwf has joined #archiveteam [13:32] *** nwf has quit IRC (Ping timeout: 633 seconds) [13:43] *** nwf has joined #archiveteam [13:49] *** WinterFox has quit IRC (Remote host closed the connection) [14:10] *** nwf has quit IRC (Ping timeout: 633 seconds) [14:12] *** nwf has joined #archiveteam [14:19] *** atomotic has joined #archiveteam [14:22] *** SN4T14 has quit IRC (Remote host closed the connection) [14:48] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [14:49] *** Wuked has joined #archiveteam [15:28] *** nwf has quit IRC (Ping timeout: 633 seconds) [15:32] *** nwf has joined #archiveteam [16:08] *** atomotic has joined #archiveteam [16:37] *** nwf has quit IRC (Ping timeout: 633 seconds) [16:51] *** nwf has joined #archiveteam [17:16] *** nwf has quit IRC (Ping timeout: 633 seconds) [17:16] *** nwf has joined #archiveteam [17:29] *** JesseW has joined #archiveteam [17:29] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [17:42] *** vitzli has quit IRC (Quit: Leaving) [17:56] *** pfallenop has quit IRC (Ping timeout: 260 seconds) [17:58] *** xXx_ndidd has joined #archiveteam [18:03] *** pfallenop has joined #archiveteam [18:08] *** nwf has quit IRC (Ping timeout: 633 seconds) [18:09] *** nwf has joined #archiveteam [18:09] *** bsmith093 has quit IRC (Ping timeout: 370 seconds) [18:10] *** ndiddy has quit IRC (Read error: Operation timed out) [18:15] *** bsmith093 has joined #archiveteam [18:18] *** xXx_ndidd is now known as ndiddy [18:29] *** pfallenop has quit IRC (Ping timeout: 260 seconds) [18:32] *** logchfoo2 starts logging #archiveteam at Sun May 08 18:32:19 2016 [18:32] *** logchfoo2 has joined #archiveteam [18:33] *** Fake-Nam1 has joined #archiveteam [18:33] *** schbirid has quit IRC (hub.efnet.us irc.Prison.NET) [18:33] *** RichardG has quit IRC (hub.efnet.us irc.Prison.NET) [18:33] *** Fake-Name has quit IRC (hub.efnet.us irc.Prison.NET) [18:33] *** vOYtEC has quit IRC (hub.efnet.us irc.Prison.NET) [18:33] *** logchfoo1 has quit IRC (hub.efnet.us irc.Prison.NET) [18:33] *** Zebranky has quit IRC (hub.efnet.us irc.Prison.NET) [18:33] *** Peetz0r has quit IRC (hub.efnet.us irc.Prison.NET) [18:33] *** Infreq has quit IRC (hub.efnet.us irc.Prison.NET) [18:33] *** sivoais has quit IRC (hub.efnet.us irc.Prison.NET) [18:33] *** achip has quit IRC (hub.efnet.us irc.Prison.NET) [18:35] *** RichardG_ has joined #archiveteam [18:37] *** Zebranky_ has joined #archiveteam [18:39] *** nwf has quit IRC (Ping timeout: 633 seconds) [18:40] *** nwf has joined #archiveteam [18:43] *** sivoais_ has joined #archiveteam [18:48] *** schbirid2 has joined #archiveteam [18:56] SketchCow: did you ever publish the script you use to add subjects to items? (I have someone who'd like to add subjects to some Italian-language books.) I checked https://github.com/internetarchive/collections-cleaners [19:09] *** pfallenop has joined #archiveteam [19:21] *** Peetz0r has joined #archiveteam [19:21] *** Wuked has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…) [19:21] *** nwf has quit IRC (Ping timeout: 633 seconds) [19:33] *** nwf has joined #archiveteam [19:53] *** BartoCH has quit IRC (Ping timeout: 260 seconds) [20:00] *** BartoCH has joined #archiveteam [20:12] *** schbirid2 has quit IRC (Quit: Leaving) [20:23] *** Ravenloft has joined #archiveteam [20:26] *** edward81 has quit IRC (Ping timeout: 492 seconds) [20:31] *** edward81 has joined #archiveteam [20:32] For arto.com: [20:32] *** ariscop has quit IRC (Read error: Operation timed out) [20:33] *** nwf has quit IRC (Ping timeout: 633 seconds) [20:34] *** Ravenloft has quit IRC (Ping timeout: 633 seconds) [20:34] *** nwf has joined #archiveteam [20:42] Videos on arto.com are streamed using rtmp [20:43] which we can't download with wget [20:43] *** JesseW has quit IRC (Ping timeout: 370 seconds) [20:43] There are a low resolution and a high resolution videos for each video [20:44] These rtmp streams are used by the flash viewer [20:44] arto.com also has some support for html5 in the source code [20:44] The html does have the high-res video as a normal http URL, so we can download that [20:45] *** edward81 has quit IRC (Read error: Operation timed out) [20:45] I haven't seen a good option though in arto.com to switch to html5, so videos might not play in the Wayback Machine [20:47] *** Honno_ has quit IRC (Read error: Operation timed out) [20:47] could we get the rtmp links out then run a script only project to get them? [20:48] The hd rtmp version is the exact same as the http video from the html5 support [20:48] 'provider': 'rtmp', [20:48] 'streamer': 'rtmp://artovideos.cloud2.artodata.com/cfx/st/', [20:48] 'file': 'data/user/video/videos/284/2841f3d3-5176-4cf1-8028-11ed777d3cc2.mp4', [20:48] 'plugins': 'hd-1', [20:48] 'hd.file': 'data/user/video/videos/0d0/0d066d26-c4b8-4da1-9545-ff5fd51bd26e.mp4', [20:48] see hd.file [20:48] html5: [20:48] { type: 'html5', config: { 'file': 'http://artovideos.cloud2.artodata.com.s3.amazonaws.com/data/user/video/videos/0d0/0d066d26-c4b8-4da1-9545-ff5fd51bd26e.mp4', 'provider': 'video' } } [20:49] we can try to grab the s3 bucket [20:49] Might be a good thing to do [20:49] I'm first starting the project though [20:49] Getting lists of URLs to ignore in now [20:50] ill have a look at the s3 bucket in the morning [20:50] Would be great! [21:16] *** ariscop has joined #archiveteam [21:24] arto-grab script are online [21:30] *** Wuked has joined #archiveteam [21:31] *** Wuked has quit IRC (Client Quit) [21:36] items for arto are loaded [21:36] arto is started! [21:36] let me know if you see anything strange [21:37] FIRST! [21:37] we're going through 6 million IDs [21:37] holy smokes, arto is fast. What service are they using? [21:38] aws? [21:39] *** metalcamp has quit IRC (Ping timeout: 244 seconds) [21:41] looks good until now, a few 403s, that get retried a few times and then ignored [21:42] do you have a log for me? [21:42] I'd like to check if it's going ok [21:43] do we have a channel for arto? [21:43] I don't think so [21:44] http://pastebin.com/QyFUWb08 [21:45] aaaaand rate limited [21:45] want a full log? [21:46] Nah, looks fine [21:46] Medowar: artodata.com seems to be coming from aws, yeah [21:47] yeah saw that, seems crazy fast [21:47] firefox starts to lag, when 6 threats are running [21:47] *** SN4T14 has joined #archiveteam [21:51] i'm seeing ~25% cpu usage with 20 threads [21:52] I'm not sure if they ban and what happends if you're banned, so let me know if you see anything [21:54] i can *try* to get myself banned if you're interested in finding out.. [21:54] nah, no need for that [21:54] chances are with autoscaling they'll start banning at some point [21:55] we'll see what happens [21:57] We'll just cost them a lot [21:57] since they are only using cloudfront/S3 and not ec2 or elb etc, they might not even have a banning system in place [21:57] HCross2: They could have expected this :) [21:58] I think S3 rate limits [21:59] they are not exposing s3, only to cloudfront, so rate limit might not trigger [22:02] *** JesseW has joined #archiveteam [22:15] *** nwf has quit IRC (Ping timeout: 633 seconds) [22:16] *** nwf has joined #archiveteam [22:20] ok, my cluster is up, checking load tomorrow and then seeing if I can fire up more workers [22:41] Do we need a lot of workers? [22:41] I'm not sure [22:41] You can fire some up, see how it goes [22:47] *** nwf has quit IRC (Ping timeout: 633 seconds) [22:48] *** nwf has joined #archiveteam [23:12] *** BlueMaxim has joined #archiveteam [23:20] *** Ravenloft has joined #archiveteam [23:28] *** RichardG_ is now known as RichardG [23:29] *** nwf has quit IRC (Ping timeout: 633 seconds) [23:29] *** nwf has joined #archiveteam [23:41] *** SN4T14 has quit IRC (Remote host closed the connection) [23:48] *** RichardG has quit IRC (Quit: Keyboard not found, press F1 to continue)