| Time |
Nickname |
Message |
|
00:02
🔗
|
|
BlueMaxim has joined #archiveteam |
|
00:25
🔗
|
|
icedice has quit IRC (Quit: Leaving) |
|
00:30
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
00:30
🔗
|
|
nwf has joined #archiveteam |
|
00:37
🔗
|
|
JesseW has joined #archiveteam |
|
00:47
🔗
|
|
TC01 has quit IRC (Read error: Operation timed out) |
|
00:57
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
01:14
🔗
|
|
Microguru has joined #archiveteam |
|
01:15
🔗
|
|
nwf has joined #archiveteam |
|
01:19
🔗
|
|
BartoCH has quit IRC (Ping timeout: 260 seconds) |
|
01:23
🔗
|
|
BlueMaxim has quit IRC (Quit: Leaving) |
|
01:23
🔗
|
|
BartoCH has joined #archiveteam |
|
01:28
🔗
|
|
JesseW has quit IRC (Ping timeout: 370 seconds) |
|
01:49
🔗
|
|
BnA-Rob1n has quit IRC (Read error: Connection reset by peer) |
|
01:49
🔗
|
|
Kazzy has quit IRC (hub.se efnet.portlane.se) |
|
01:49
🔗
|
|
Igloo has quit IRC (hub.se efnet.portlane.se) |
|
01:49
🔗
|
|
Fletcher_ has quit IRC (hub.se efnet.portlane.se) |
|
01:49
🔗
|
|
koon has quit IRC (hub.se efnet.portlane.se) |
|
01:49
🔗
|
|
superkuh has joined #archiveteam |
|
01:50
🔗
|
|
Sk1d has quit IRC (Ping timeout: 194 seconds) |
|
01:51
🔗
|
|
Nemo_bis has quit IRC (Ping timeout: 244 seconds) |
|
02:06
🔗
|
|
TC01 has joined #archiveteam |
|
02:11
🔗
|
|
BnA-Rob1n has joined #archiveteam |
|
02:11
🔗
|
|
Nemo_bis has joined #archiveteam |
|
02:14
🔗
|
|
superkuh has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
BartoCH has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
hive-mind has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
zhongfu has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
philpem has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
Start has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
r3c0d3x has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
lesderid has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
sigkell has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
pfallenop has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
Meroje has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
Coderjoe has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
bauruine has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
Jon has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
FalconK has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
balrog has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
d_rebel has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
zerkalo has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
SirCmpwn has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
szalwia has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
antonizoo has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
Atluxity has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
Sanqui has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
hictooth has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
davidar has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
Rickster has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
Muad-Dib has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
Famicoman has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
_desu___ has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
pikhq has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
bai has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
dan- has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
kevin has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
karissa__ has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
JSharp___ has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
victor has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
VonGuard has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
Vito` has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
Boltsie has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
TheKiwi has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
HCross2 has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
Ctrl-S___ has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
deathy has quit IRC (hub.se efnet.port80.se) |
|
02:14
🔗
|
|
johtso has quit IRC (hub.se efnet.port80.se) |
|
02:53
🔗
|
|
WinterFox has joined #archiveteam |
|
03:01
🔗
|
|
Ravenloft has joined #archiveteam |
|
03:06
🔗
|
|
balrog has joined #archiveteam |
|
03:06
🔗
|
|
swebb sets mode: +o balrog |
|
03:07
🔗
|
|
JesseW has joined #archiveteam |
|
03:16
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
03:20
🔗
|
|
nwf has joined #archiveteam |
|
03:40
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
03:40
🔗
|
|
nwf has joined #archiveteam |
|
03:41
🔗
|
|
JesseW has quit IRC (Ping timeout: 370 seconds) |
|
03:50
🔗
|
|
JesseW has joined #archiveteam |
|
04:19
🔗
|
|
vitzli has joined #archiveteam |
|
04:22
🔗
|
|
BlueMaxim has joined #archiveteam |
|
04:41
🔗
|
|
zhongfu has joined #archiveteam |
|
04:41
🔗
|
|
superkuh has joined #archiveteam |
|
04:41
🔗
|
|
BartoCH has joined #archiveteam |
|
04:41
🔗
|
|
hive-mind has joined #archiveteam |
|
04:41
🔗
|
|
philpem has joined #archiveteam |
|
04:41
🔗
|
|
Start has joined #archiveteam |
|
04:41
🔗
|
|
r3c0d3x has joined #archiveteam |
|
04:41
🔗
|
|
lesderid has joined #archiveteam |
|
04:41
🔗
|
|
sigkell has joined #archiveteam |
|
04:41
🔗
|
|
pfallenop has joined #archiveteam |
|
04:41
🔗
|
|
Meroje has joined #archiveteam |
|
04:41
🔗
|
|
Coderjoe has joined #archiveteam |
|
04:41
🔗
|
|
bauruine has joined #archiveteam |
|
04:41
🔗
|
|
Jon has joined #archiveteam |
|
04:41
🔗
|
|
FalconK has joined #archiveteam |
|
04:41
🔗
|
|
d_rebel has joined #archiveteam |
|
04:41
🔗
|
|
zerkalo has joined #archiveteam |
|
04:41
🔗
|
|
SirCmpwn has joined #archiveteam |
|
04:41
🔗
|
|
szalwia has joined #archiveteam |
|
04:41
🔗
|
|
antonizoo has joined #archiveteam |
|
04:41
🔗
|
|
Atluxity has joined #archiveteam |
|
04:41
🔗
|
|
Sanqui has joined #archiveteam |
|
04:41
🔗
|
|
hictooth has joined #archiveteam |
|
04:41
🔗
|
|
davidar has joined #archiveteam |
|
04:41
🔗
|
|
Rickster has joined #archiveteam |
|
04:41
🔗
|
|
Muad-Dib has joined #archiveteam |
|
04:41
🔗
|
|
Famicoman has joined #archiveteam |
|
04:41
🔗
|
|
_desu___ has joined #archiveteam |
|
04:41
🔗
|
|
pikhq has joined #archiveteam |
|
04:41
🔗
|
|
bai has joined #archiveteam |
|
04:41
🔗
|
|
dan- has joined #archiveteam |
|
04:41
🔗
|
|
kevin has joined #archiveteam |
|
04:41
🔗
|
|
karissa__ has joined #archiveteam |
|
04:41
🔗
|
|
JSharp___ has joined #archiveteam |
|
04:41
🔗
|
|
victor has joined #archiveteam |
|
04:41
🔗
|
|
Ctrl-S___ has joined #archiveteam |
|
04:41
🔗
|
|
VonGuard has joined #archiveteam |
|
04:41
🔗
|
|
Vito` has joined #archiveteam |
|
04:41
🔗
|
|
HCross2 has joined #archiveteam |
|
04:41
🔗
|
|
Boltsie has joined #archiveteam |
|
04:41
🔗
|
|
TheKiwi has joined #archiveteam |
|
04:41
🔗
|
|
deathy has joined #archiveteam |
|
04:41
🔗
|
|
johtso has joined #archiveteam |
|
04:41
🔗
|
|
efnet.port80.se sets mode: +oo Atluxity HCross2 |
|
04:41
🔗
|
|
swebb sets mode: +o Atluxity |
|
04:42
🔗
|
|
Kazzy has joined #archiveteam |
|
04:42
🔗
|
|
Igloo has joined #archiveteam |
|
04:42
🔗
|
|
Fletcher_ has joined #archiveteam |
|
04:42
🔗
|
|
koon has joined #archiveteam |
|
04:43
🔗
|
|
Sk1d has joined #archiveteam |
|
05:07
🔗
|
|
ralphdnak has joined #archiveteam |
|
05:18
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
05:21
🔗
|
|
nwf has joined #archiveteam |
|
05:27
🔗
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
05:31
🔗
|
|
dashcloud has joined #archiveteam |
|
05:34
🔗
|
|
edward81 has joined #archiveteam |
|
05:40
🔗
|
|
Microguru has quit IRC (Quit: Microguru) |
|
05:52
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
05:58
🔗
|
|
metalcamp has joined #archiveteam |
|
05:59
🔗
|
|
nwf has joined #archiveteam |
|
06:01
🔗
|
|
VADemon has quit IRC (Quit: left4dead) |
|
06:14
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
06:15
🔗
|
|
nwf has joined #archiveteam |
|
06:25
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
06:32
🔗
|
JesseW |
I don't see arto in the warrior tracker. arkiver -- how are you running it? |
|
06:41
🔗
|
|
nwf has joined #archiveteam |
|
06:45
🔗
|
|
sHATNER has joined #archiveteam |
|
07:14
🔗
|
|
JesseW has quit IRC (Ping timeout: 370 seconds) |
|
07:14
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
07:23
🔗
|
|
nwf has joined #archiveteam |
|
07:49
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
07:50
🔗
|
|
nwf has joined #archiveteam |
|
08:42
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
08:43
🔗
|
|
nwf has joined #archiveteam |
|
08:54
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
08:54
🔗
|
|
pfallenop has quit IRC (Ping timeout: 260 seconds) |
|
09:06
🔗
|
|
r3c0d3x has quit IRC (Ping timeout: 260 seconds) |
|
09:06
🔗
|
|
r3c0d3x has joined #archiveteam |
|
09:31
🔗
|
|
nwf has joined #archiveteam |
|
09:56
🔗
|
|
Ravenloft has quit IRC (Read error: Connection reset by peer) |
|
09:56
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
09:58
🔗
|
|
nwf has joined #archiveteam |
|
10:09
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
10:10
🔗
|
|
nwf has joined #archiveteam |
|
10:13
🔗
|
|
ralphdnak has quit IRC (Read error: Operation timed out) |
|
10:20
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
10:21
🔗
|
|
nwf has joined #archiveteam |
|
10:33
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
10:38
🔗
|
|
nwf has joined #archiveteam |
|
10:54
🔗
|
|
pfallenop has joined #archiveteam |
|
11:00
🔗
|
|
luckcolor has quit IRC (http://quassel-irc.org - Chat comfortably. Anywhere.) |
|
11:00
🔗
|
|
luckcolor has joined #archiveteam |
|
11:02
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
11:02
🔗
|
|
luckcolor has quit IRC (Client Quit) |
|
11:02
🔗
|
|
luckcolor has joined #archiveteam |
|
11:03
🔗
|
|
luckcolor has quit IRC (Client Quit) |
|
11:03
🔗
|
|
luckcolor has joined #archiveteam |
|
11:05
🔗
|
|
luckcolor has quit IRC (Client Quit) |
|
11:07
🔗
|
|
luckcolor has joined #archiveteam |
|
11:07
🔗
|
|
luckcolor has quit IRC (Client Quit) |
|
11:07
🔗
|
|
luckcolor has joined #archiveteam |
|
11:11
🔗
|
|
nwf has joined #archiveteam |
|
11:16
🔗
|
|
BlueMaxim has quit IRC (Quit: Leaving) |
|
11:42
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
11:42
🔗
|
|
nwf has joined #archiveteam |
|
11:49
🔗
|
arkiver |
JesseW: Scripts are ready, I didn't start it yet |
|
11:49
🔗
|
arkiver |
I try to never start project only an hour before I go to bed |
|
11:49
🔗
|
arkiver |
Often we need to make limiting changes and some small script fixes/additions |
|
11:52
🔗
|
luckcolor |
https://github.com/bevacqua/shots |
|
11:52
🔗
|
luckcolor |
Animated gif of archive.org website history |
|
11:54
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
12:01
🔗
|
|
nwf has joined #archiveteam |
|
12:21
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
12:21
🔗
|
|
nwf has joined #archiveteam |
|
13:32
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
13:43
🔗
|
|
nwf has joined #archiveteam |
|
13:49
🔗
|
|
WinterFox has quit IRC (Remote host closed the connection) |
|
14:10
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
14:12
🔗
|
|
nwf has joined #archiveteam |
|
14:19
🔗
|
|
atomotic has joined #archiveteam |
|
14:22
🔗
|
|
SN4T14 has quit IRC (Remote host closed the connection) |
|
14:48
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
|
14:49
🔗
|
|
Wuked has joined #archiveteam |
|
15:28
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
15:32
🔗
|
|
nwf has joined #archiveteam |
|
16:08
🔗
|
|
atomotic has joined #archiveteam |
|
16:37
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
16:51
🔗
|
|
nwf has joined #archiveteam |
|
17:16
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
17:16
🔗
|
|
nwf has joined #archiveteam |
|
17:29
🔗
|
|
JesseW has joined #archiveteam |
|
17:29
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
|
17:42
🔗
|
|
vitzli has quit IRC (Quit: Leaving) |
|
17:56
🔗
|
|
pfallenop has quit IRC (Ping timeout: 260 seconds) |
|
17:58
🔗
|
|
xXx_ndidd has joined #archiveteam |
|
18:03
🔗
|
|
pfallenop has joined #archiveteam |
|
18:08
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
18:09
🔗
|
|
nwf has joined #archiveteam |
|
18:09
🔗
|
|
bsmith093 has quit IRC (Ping timeout: 370 seconds) |
|
18:10
🔗
|
|
ndiddy has quit IRC (Read error: Operation timed out) |
|
18:15
🔗
|
|
bsmith093 has joined #archiveteam |
|
18:18
🔗
|
|
xXx_ndidd is now known as ndiddy |
|
18:29
🔗
|
|
pfallenop has quit IRC (Ping timeout: 260 seconds) |
|
18:32
🔗
|
|
logchfoo2 starts logging #archiveteam at Sun May 08 18:32:19 2016 |
|
18:32
🔗
|
|
logchfoo2 has joined #archiveteam |
|
18:33
🔗
|
|
Fake-Nam1 has joined #archiveteam |
|
18:33
🔗
|
|
schbirid has quit IRC (hub.efnet.us irc.Prison.NET) |
|
18:33
🔗
|
|
RichardG has quit IRC (hub.efnet.us irc.Prison.NET) |
|
18:33
🔗
|
|
Fake-Name has quit IRC (hub.efnet.us irc.Prison.NET) |
|
18:33
🔗
|
|
vOYtEC has quit IRC (hub.efnet.us irc.Prison.NET) |
|
18:33
🔗
|
|
logchfoo1 has quit IRC (hub.efnet.us irc.Prison.NET) |
|
18:33
🔗
|
|
Zebranky has quit IRC (hub.efnet.us irc.Prison.NET) |
|
18:33
🔗
|
|
Peetz0r has quit IRC (hub.efnet.us irc.Prison.NET) |
|
18:33
🔗
|
|
Infreq has quit IRC (hub.efnet.us irc.Prison.NET) |
|
18:33
🔗
|
|
sivoais has quit IRC (hub.efnet.us irc.Prison.NET) |
|
18:33
🔗
|
|
achip has quit IRC (hub.efnet.us irc.Prison.NET) |
|
18:35
🔗
|
|
RichardG_ has joined #archiveteam |
|
18:37
🔗
|
|
Zebranky_ has joined #archiveteam |
|
18:39
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
18:40
🔗
|
|
nwf has joined #archiveteam |
|
18:43
🔗
|
|
sivoais_ has joined #archiveteam |
|
18:48
🔗
|
|
schbirid2 has joined #archiveteam |
|
18:56
🔗
|
Nemo_bis |
SketchCow: did you ever publish the script you use to add subjects to items? (I have someone who'd like to add subjects to some Italian-language books.) I checked https://github.com/internetarchive/collections-cleaners |
|
19:09
🔗
|
|
pfallenop has joined #archiveteam |
|
19:21
🔗
|
|
Peetz0r has joined #archiveteam |
|
19:21
🔗
|
|
Wuked has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…) |
|
19:21
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
19:33
🔗
|
|
nwf has joined #archiveteam |
|
19:53
🔗
|
|
BartoCH has quit IRC (Ping timeout: 260 seconds) |
|
20:00
🔗
|
|
BartoCH has joined #archiveteam |
|
20:12
🔗
|
|
schbirid2 has quit IRC (Quit: Leaving) |
|
20:23
🔗
|
|
Ravenloft has joined #archiveteam |
|
20:26
🔗
|
|
edward81 has quit IRC (Ping timeout: 492 seconds) |
|
20:31
🔗
|
|
edward81 has joined #archiveteam |
|
20:32
🔗
|
arkiver |
For arto.com: |
|
20:32
🔗
|
|
ariscop has quit IRC (Read error: Operation timed out) |
|
20:33
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
20:34
🔗
|
|
Ravenloft has quit IRC (Ping timeout: 633 seconds) |
|
20:34
🔗
|
|
nwf has joined #archiveteam |
|
20:42
🔗
|
arkiver |
Videos on arto.com are streamed using rtmp |
|
20:43
🔗
|
arkiver |
which we can't download with wget |
|
20:43
🔗
|
|
JesseW has quit IRC (Ping timeout: 370 seconds) |
|
20:43
🔗
|
arkiver |
There are a low resolution and a high resolution videos for each video |
|
20:44
🔗
|
arkiver |
These rtmp streams are used by the flash viewer |
|
20:44
🔗
|
arkiver |
arto.com also has some support for html5 in the source code |
|
20:44
🔗
|
arkiver |
The html does have the high-res video as a normal http URL, so we can download that |
|
20:45
🔗
|
|
edward81 has quit IRC (Read error: Operation timed out) |
|
20:45
🔗
|
arkiver |
I haven't seen a good option though in arto.com to switch to html5, so videos might not play in the Wayback Machine |
|
20:47
🔗
|
|
Honno_ has quit IRC (Read error: Operation timed out) |
|
20:47
🔗
|
HCross |
could we get the rtmp links out then run a script only project to get them? |
|
20:48
🔗
|
arkiver |
The hd rtmp version is the exact same as the http video from the html5 support |
|
20:48
🔗
|
arkiver |
'provider': 'rtmp', |
|
20:48
🔗
|
arkiver |
'streamer': 'rtmp://artovideos.cloud2.artodata.com/cfx/st/', |
|
20:48
🔗
|
arkiver |
'file': 'data/user/video/videos/284/2841f3d3-5176-4cf1-8028-11ed777d3cc2.mp4', |
|
20:48
🔗
|
arkiver |
'plugins': 'hd-1', |
|
20:48
🔗
|
arkiver |
'hd.file': 'data/user/video/videos/0d0/0d066d26-c4b8-4da1-9545-ff5fd51bd26e.mp4', |
|
20:48
🔗
|
arkiver |
see hd.file |
|
20:48
🔗
|
arkiver |
html5: |
|
20:48
🔗
|
arkiver |
{ type: 'html5', config: { 'file': 'http://artovideos.cloud2.artodata.com.s3.amazonaws.com/data/user/video/videos/0d0/0d066d26-c4b8-4da1-9545-ff5fd51bd26e.mp4', 'provider': 'video' } } |
|
20:49
🔗
|
midas |
we can try to grab the s3 bucket |
|
20:49
🔗
|
arkiver |
Might be a good thing to do |
|
20:49
🔗
|
arkiver |
I'm first starting the project though |
|
20:49
🔗
|
arkiver |
Getting lists of URLs to ignore in now |
|
20:50
🔗
|
midas |
ill have a look at the s3 bucket in the morning |
|
20:50
🔗
|
arkiver |
Would be great! |
|
21:16
🔗
|
|
ariscop has joined #archiveteam |
|
21:24
🔗
|
arkiver |
arto-grab script are online |
|
21:30
🔗
|
|
Wuked has joined #archiveteam |
|
21:31
🔗
|
|
Wuked has quit IRC (Client Quit) |
|
21:36
🔗
|
arkiver |
items for arto are loaded |
|
21:36
🔗
|
arkiver |
arto is started! |
|
21:36
🔗
|
arkiver |
let me know if you see anything strange |
|
21:37
🔗
|
Medowar |
FIRST! |
|
21:37
🔗
|
arkiver |
we're going through 6 million IDs |
|
21:37
🔗
|
Medowar |
holy smokes, arto is fast. What service are they using? |
|
21:38
🔗
|
Medowar |
aws? |
|
21:39
🔗
|
|
metalcamp has quit IRC (Ping timeout: 244 seconds) |
|
21:41
🔗
|
Medowar |
looks good until now, a few 403s, that get retried a few times and then ignored |
|
21:42
🔗
|
arkiver |
do you have a log for me? |
|
21:42
🔗
|
arkiver |
I'd like to check if it's going ok |
|
21:43
🔗
|
Kazzy |
do we have a channel for arto? |
|
21:43
🔗
|
arkiver |
I don't think so |
|
21:44
🔗
|
Medowar |
http://pastebin.com/QyFUWb08 |
|
21:45
🔗
|
Medowar |
aaaaand rate limited |
|
21:45
🔗
|
Medowar |
want a full log? |
|
21:46
🔗
|
arkiver |
Nah, looks fine |
|
21:46
🔗
|
Kazzy |
Medowar: artodata.com seems to be coming from aws, yeah |
|
21:47
🔗
|
Medowar |
yeah saw that, seems crazy fast |
|
21:47
🔗
|
Medowar |
firefox starts to lag, when 6 threats are running |
|
21:47
🔗
|
|
SN4T14 has joined #archiveteam |
|
21:51
🔗
|
Kazzy |
i'm seeing ~25% cpu usage with 20 threads |
|
21:52
🔗
|
arkiver |
I'm not sure if they ban and what happends if you're banned, so let me know if you see anything |
|
21:54
🔗
|
Kazzy |
i can *try* to get myself banned if you're interested in finding out.. |
|
21:54
🔗
|
arkiver |
nah, no need for that |
|
21:54
🔗
|
Kazzy |
chances are with autoscaling they'll start banning at some point |
|
21:55
🔗
|
arkiver |
we'll see what happens |
|
21:57
🔗
|
HCross2 |
We'll just cost them a lot |
|
21:57
🔗
|
Medowar |
since they are only using cloudfront/S3 and not ec2 or elb etc, they might not even have a banning system in place |
|
21:57
🔗
|
arkiver |
HCross2: They could have expected this :) |
|
21:58
🔗
|
HCross2 |
I think S3 rate limits |
|
21:59
🔗
|
Medowar |
they are not exposing s3, only to cloudfront, so rate limit might not trigger |
|
22:02
🔗
|
|
JesseW has joined #archiveteam |
|
22:15
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
22:16
🔗
|
|
nwf has joined #archiveteam |
|
22:20
🔗
|
Medowar |
ok, my cluster is up, checking load tomorrow and then seeing if I can fire up more workers |
|
22:41
🔗
|
HCross2 |
Do we need a lot of workers? |
|
22:41
🔗
|
arkiver |
I'm not sure |
|
22:41
🔗
|
arkiver |
You can fire some up, see how it goes |
|
22:47
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
22:48
🔗
|
|
nwf has joined #archiveteam |
|
23:12
🔗
|
|
BlueMaxim has joined #archiveteam |
|
23:20
🔗
|
|
Ravenloft has joined #archiveteam |
|
23:28
🔗
|
|
RichardG_ is now known as RichardG |
|
23:29
🔗
|
|
nwf has quit IRC (Ping timeout: 633 seconds) |
|
23:29
🔗
|
|
nwf has joined #archiveteam |
|
23:41
🔗
|
|
SN4T14 has quit IRC (Remote host closed the connection) |
|
23:48
🔗
|
|
RichardG has quit IRC (Quit: Keyboard not found, press F1 to continue) |