| Time |
Nickname |
Message |
|
00:04
🔗
|
|
odemg has joined #archiveteam |
|
00:06
🔗
|
VADemon |
TMM: meanwhile your robots.txt does not define it (on the main domain)? |
|
00:07
🔗
|
TMM |
VADemon, I only just added one today as a kind of kneejerk to the issue |
|
00:07
🔗
|
TMM |
VADemon, if archivebot had supported it I'd have done something a little nicer |
|
00:08
🔗
|
VADemon |
Archivebot jobs are run manually by people issuing the command. Can someone add a reminder to #archivebot to be merciful with sleep times for crawling? |
|
00:09
🔗
|
|
odemg has quit IRC (Remote host closed the connection) |
|
00:09
🔗
|
TMM |
VADemon, I realize they are ran manually, but if there's a robots.txt with a crawl-delay it'd be nice if it could honor it |
|
00:12
🔗
|
xmc |
default crawl params for archivebot is 3 threads, fetching links with an aggregate target of 250-375 between requests on average |
|
00:12
🔗
|
xmc |
er, ms |
|
00:12
🔗
|
TMM |
yeah, some of those changesets take 20+ seconds to generate |
|
00:12
🔗
|
TMM |
someone put a whole firefox sourcetree on there :) |
|
00:13
🔗
|
xmc |
that'll do it |
|
00:13
🔗
|
xmc |
most websites are less cpu-intensive than yours |
|
00:21
🔗
|
|
Stilett0 has joined #archiveteam |
|
00:39
🔗
|
|
travisbel has joined #archiveteam |
|
00:40
🔗
|
travisbel |
Anyone here familiar with the IMDB archive (https://archive.org/details/archiveteam_imdb)? |
|
00:46
🔗
|
|
pnJay has joined #archiveteam |
|
00:46
🔗
|
|
travisbel has quit IRC (Quit: Page closed) |
|
00:46
🔗
|
|
travisbel has joined #archiveteam |
|
00:46
🔗
|
|
pnJay has quit IRC (Client Quit) |
|
00:46
🔗
|
|
travisbel has quit IRC (Client Quit) |
|
00:51
🔗
|
|
pnJay has joined #archiveteam |
|
00:51
🔗
|
|
odemg has joined #archiveteam |
|
00:54
🔗
|
SketchCow |
Travis, you are going to get into so much trouble, you know that right |
|
00:56
🔗
|
|
Aranje has quit IRC (Ping timeout: 506 seconds) |
|
00:58
🔗
|
SketchCow |
Do you want to know your next 60 days or do you want it spoiled |
|
01:13
🔗
|
|
lesderid has quit IRC (Ping timeout: 260 seconds) |
|
01:14
🔗
|
|
Ymgve has quit IRC (Ping timeout: 506 seconds) |
|
01:16
🔗
|
|
Silvan has quit IRC (Read error: Connection reset by peer) |
|
01:18
🔗
|
|
SilSte has joined #archiveteam |
|
01:19
🔗
|
|
Meeh has quit IRC (Remote host closed the connection) |
|
01:19
🔗
|
|
Meeh has joined #archiveteam |
|
01:20
🔗
|
|
Meeh has quit IRC (Remote host closed the connection) |
|
01:20
🔗
|
|
Meeh has joined #archiveteam |
|
01:22
🔗
|
|
Meeh has quit IRC (Remote host closed the connection) |
|
01:22
🔗
|
|
Meeh has joined #archiveteam |
|
01:22
🔗
|
|
icedice has quit IRC (Quit: Leaving) |
|
01:23
🔗
|
|
Meeh has quit IRC (Remote host closed the connection) |
|
01:23
🔗
|
|
Meeh has joined #archiveteam |
|
01:24
🔗
|
|
Meeh has quit IRC (Remote host closed the connection) |
|
01:29
🔗
|
|
lesderid has joined #archiveteam |
|
01:35
🔗
|
|
VADemon has quit IRC (Quit: left4dead) |
|
01:45
🔗
|
|
Meeh has joined #archiveteam |
|
01:45
🔗
|
|
Ymgve has joined #archiveteam |
|
01:47
🔗
|
|
Meeh has quit IRC (Remote host closed the connection) |
|
01:47
🔗
|
|
Meeh has joined #archiveteam |
|
01:47
🔗
|
|
Meeh has quit IRC (Remote host closed the connection) |
|
01:47
🔗
|
|
Meeh has joined #archiveteam |
|
01:48
🔗
|
|
Meeh has quit IRC (Remote host closed the connection) |
|
01:50
🔗
|
|
Meeh has joined #archiveteam |
|
01:50
🔗
|
|
Meeh has quit IRC (Remote host closed the connection) |
|
01:50
🔗
|
|
Meeh has joined #archiveteam |
|
01:51
🔗
|
|
Meeh has quit IRC (Remote host closed the connection) |
|
01:54
🔗
|
|
Meeh has joined #archiveteam |
|
01:54
🔗
|
|
Meeh has quit IRC (Remote host closed the connection) |
|
01:55
🔗
|
|
Meeh has joined #archiveteam |
|
02:05
🔗
|
|
j08nY has quit IRC (Quit: Leaving) |
|
02:08
🔗
|
|
HCross2 has quit IRC (Quit: Connection closed for inactivity) |
|
02:09
🔗
|
|
pnJay has quit IRC (Leaving) |
|
02:13
🔗
|
|
namespace has joined #archiveteam |
|
02:16
🔗
|
|
Stilett0 has quit IRC (Read error: Operation timed out) |
|
02:23
🔗
|
|
ndiddy has joined #archiveteam |
|
02:33
🔗
|
|
TMM has quit IRC (Ping timeout: 255 seconds) |
|
02:35
🔗
|
|
TMM has joined #archiveteam |
|
02:49
🔗
|
|
TMM has quit IRC (Ping timeout: 255 seconds) |
|
02:50
🔗
|
|
TMM has joined #archiveteam |
|
03:11
🔗
|
|
Stilett0 has joined #archiveteam |
|
04:18
🔗
|
|
ndiddy has quit IRC (Read error: Connection reset by peer) |
|
04:42
🔗
|
|
pizzaiolo has left |
|
04:42
🔗
|
|
maelstrom has quit IRC (Quit: Leaving) |
|
05:08
🔗
|
|
Sk1d has quit IRC (Ping timeout: 194 seconds) |
|
05:18
🔗
|
|
squires has joined #archiveteam |
|
05:22
🔗
|
|
HCross2 has joined #archiveteam |
|
05:49
🔗
|
|
topdownji has quit IRC (Remote host closed the connection) |
|
05:50
🔗
|
|
topdownji has joined #archiveteam |
|
06:35
🔗
|
SketchCow |
P.S. FOS has been moving (not WARCing, moving) items out of the imdb dropbox for a week straight |
|
07:01
🔗
|
|
SilSte has quit IRC (Read error: Connection reset by peer) |
|
07:03
🔗
|
|
SilSte has joined #archiveteam |
|
07:06
🔗
|
|
kris33 has joined #archiveteam |
|
07:34
🔗
|
|
no2pencil has joined #archiveteam |
|
07:34
🔗
|
no2pencil |
SketchCow: sending you good thoughts man, read about your ordeal on Twitter, sorry man. |
|
07:34
🔗
|
|
schbirid has joined #archiveteam |
|
07:54
🔗
|
|
QBcrusher has quit IRC (Read error: Connection reset by peer) |
|
08:04
🔗
|
|
odemg has quit IRC (Remote host closed the connection) |
|
08:05
🔗
|
|
kyounko has quit IRC (Ping timeout: 246 seconds) |
|
08:06
🔗
|
SketchCow |
I'm fiiiiine |
|
08:26
🔗
|
|
BlueMaxim has quit IRC (Read error: Operation timed out) |
|
08:30
🔗
|
|
ravetcofx has quit IRC (Ping timeout: 245 seconds) |
|
09:07
🔗
|
|
atomotic has joined #archiveteam |
|
09:09
🔗
|
|
bwn has quit IRC (Read error: Operation timed out) |
|
09:12
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
|
09:31
🔗
|
|
TMM has quit IRC (Quit: Ex-Chat) |
|
10:18
🔗
|
|
ZexaronS has quit IRC (Quit: Leaving) |
|
10:24
🔗
|
|
TMM has joined #archiveteam |
|
10:25
🔗
|
|
jspiros has quit IRC (Read error: Operation timed out) |
|
10:26
🔗
|
|
SadDM has quit IRC (Read error: Operation timed out) |
|
10:50
🔗
|
|
gui7 has joined #archiveteam |
|
10:50
🔗
|
gui7 |
can we make sure this stuff is saved? :) https://www.reddit.com/r/learnprogramming/comments/5x3ihz/uc_berkeley_to_remove_free_lecture_videos_from/?st=iztp3ugy&sh=3e8552dc |
|
10:51
🔗
|
tobbez |
gui7: #berklost |
|
10:52
🔗
|
gui7 |
thanks |
|
10:53
🔗
|
|
atomotic has joined #archiveteam |
|
11:00
🔗
|
|
eightfold has joined #archiveteam |
|
11:04
🔗
|
|
eightfold has quit IRC (Ping timeout: 260 seconds) |
|
11:08
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
|
11:26
🔗
|
|
jspiros has joined #archiveteam |
|
11:26
🔗
|
|
SadDM has joined #archiveteam |
|
11:26
🔗
|
|
swebb sets mode: +o SadDM |
|
11:30
🔗
|
|
pizzaiolo has joined #archiveteam |
|
11:39
🔗
|
|
atomotic has joined #archiveteam |
|
11:43
🔗
|
|
atomotic has quit IRC (Client Quit) |
|
12:00
🔗
|
|
bwn has joined #archiveteam |
|
12:23
🔗
|
|
TMM has quit IRC (Quit: Ex-Chat) |
|
12:24
🔗
|
|
LastNinja has joined #archiveteam |
|
12:25
🔗
|
|
icedice has joined #archiveteam |
|
12:32
🔗
|
|
bwn has quit IRC (Read error: Operation timed out) |
|
12:40
🔗
|
|
Silvan has joined #archiveteam |
|
12:41
🔗
|
|
SilSte has quit IRC (Read error: Operation timed out) |
|
12:52
🔗
|
|
bwn has joined #archiveteam |
|
13:15
🔗
|
|
gibigiana has quit IRC (Ping timeout: 370 seconds) |
|
13:15
🔗
|
|
gibigiana has joined #archiveteam |
|
13:17
🔗
|
|
bwn has quit IRC (Read error: Operation timed out) |
|
13:44
🔗
|
|
khaoohs has joined #archiveteam |
|
14:09
🔗
|
|
pnJay has joined #archiveteam |
|
14:17
🔗
|
|
Gfy has quit IRC (Ping timeout: 250 seconds) |
|
14:54
🔗
|
|
yonatan has joined #archiveteam |
|
15:20
🔗
|
|
Boppen has joined #archiveteam |
|
15:57
🔗
|
|
C4K3 has joined #archiveteam |
|
15:58
🔗
|
|
yonatan has quit IRC (Quit: Page closed) |
|
16:00
🔗
|
|
C4K3_ has quit IRC (Ping timeout: 250 seconds) |
|
16:18
🔗
|
|
gui7 has quit IRC (Ping timeout: 250 seconds) |
|
16:31
🔗
|
|
Gfy has joined #archiveteam |
|
16:37
🔗
|
|
bwn has joined #archiveteam |
|
17:03
🔗
|
|
Aranje has joined #archiveteam |
|
17:04
🔗
|
|
gui7 has joined #archiveteam |
|
17:10
🔗
|
|
fie has quit IRC (Ping timeout: 244 seconds) |
|
17:15
🔗
|
|
mls has joined #archiveteam |
|
17:28
🔗
|
|
fie has joined #archiveteam |
|
17:34
🔗
|
|
Smiley has quit IRC (Quit: Lost terminal) |
|
17:34
🔗
|
|
ravetcofx has joined #archiveteam |
|
17:34
🔗
|
|
Smiley has joined #archiveteam |
|
17:48
🔗
|
|
odemg has joined #archiveteam |
|
18:26
🔗
|
mundus201 |
Hi, I know this isnt the right channel but im not getting much support elsewhere, Im tryinng to run the warrior with docker but docker is saying that it cannot connect to the daemon. Ive tried restarting the docker service, reinstalling it, killing it, and deleting the pid but I cant seem to get it to work, this is what it says when I try docker daemon -D, http://pastebin.com/raw/B7wCC9r0 |
|
18:29
🔗
|
joepie91 |
mundus201: sounds like a Docker problem |
|
18:30
🔗
|
mundus201 |
hmm looks like when I stopped the service I was able to start the daemon |
|
18:31
🔗
|
mundus201 |
but still have issues connecting |
|
18:50
🔗
|
|
VADemon has joined #archiveteam |
|
18:53
🔗
|
mundus201 |
yeah im dumb, I add to add my user to the docker group |
|
19:04
🔗
|
macker |
docker needs to run as root? |
|
19:05
🔗
|
mundus201 |
Yeah |
|
19:05
🔗
|
macker |
guess it makes sense, Ive only p[layed around |
|
19:05
🔗
|
macker |
sorry I cant help |
|
19:05
🔗
|
mundus201 |
Ive got it fixed now |
|
19:05
🔗
|
mundus201 |
now im having issues with my warrior |
|
19:05
🔗
|
mundus201 |
"Error communicating with tracker: 404 Client Error: No items available currently. Don't worry; more items will be made soon. Try again later. for url: http://tracker.archiveteam.org:1337/api/get." |
|
19:22
🔗
|
|
Morbus has joined #archiveteam |
|
19:24
🔗
|
ThisAsYou |
mundus201: I usually run docker with sudo unless it's a dev machine. |
|
19:37
🔗
|
mls |
mundus201: urlteam can be like that, items will come your way |
|
19:37
🔗
|
mundus201 |
ah okay |
|
19:38
🔗
|
mundus201 |
This is my first time running the warrior |
|
20:07
🔗
|
|
QBcrusher has joined #archiveteam |
|
20:19
🔗
|
icedice |
https://torrentfreak.com/torrent-legend-mininova-will-shut-down-for-good-170226/ |
|
20:20
🔗
|
icedice |
^ any plans for this? |
|
20:33
🔗
|
|
redlob has quit IRC (Read error: Operation timed out) |
|
20:34
🔗
|
|
redlob has joined #archiveteam |
|
20:35
🔗
|
nightpool |
icedice: we did some archivebot grabs of the public content. |
|
20:35
🔗
|
icedice |
Ok |
|
20:35
🔗
|
nightpool |
Althought it doesn't look like they went all the way through.... hm. |
|
20:37
🔗
|
|
peep_ has joined #archiveteam |
|
20:37
🔗
|
|
peep_ has quit IRC (Client Quit) |
|
20:44
🔗
|
|
anonymoos has joined #archiveteam |
|
20:46
🔗
|
nightpool |
icedice: looking at the logs, we were IP banned for requesting too fast, so we're re-running the grab much slower, and we'll see how it goes |
|
20:47
🔗
|
icedice |
ok |
|
20:56
🔗
|
|
bRick5772 has joined #archiveteam |
|
21:18
🔗
|
|
redlob has quit IRC (Read error: Operation timed out) |
|
21:20
🔗
|
|
REiN^ has quit IRC (Max SendQ exceeded) |
|
21:20
🔗
|
|
REiN^ has joined #archiveteam |
|
21:25
🔗
|
|
redlob has joined #archiveteam |
|
21:33
🔗
|
|
icedice has quit IRC (Read error: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac) |
|
21:34
🔗
|
|
icedice has joined #archiveteam |
|
21:36
🔗
|
|
j08nY has joined #archiveteam |
|
21:45
🔗
|
|
odemg has quit IRC (Remote host closed the connection) |
|
21:45
🔗
|
|
odemg has joined #archiveteam |
|
22:02
🔗
|
|
pnJay has quit IRC (Quit: Page closed) |
|
22:07
🔗
|
|
REiN^ has quit IRC (Max SendQ exceeded) |
|
22:07
🔗
|
|
_desu___ has quit IRC (Max SendQ exceeded) |
|
22:11
🔗
|
|
REiN^ has joined #archiveteam |
|
22:16
🔗
|
|
pnJay has joined #archiveteam |
|
23:01
🔗
|
|
pizzaiolo has quit IRC (Remote host closed the connection) |
|
23:05
🔗
|
|
bRick5772 has quit IRC (Quit: Leaving.) |
|
23:10
🔗
|
|
Ravenloft has joined #archiveteam |
|
23:13
🔗
|
|
dashcloud has quit IRC (Quit: No Ping reply in 180 seconds.) |
|
23:15
🔗
|
|
dashcloud has joined #archiveteam |
|
23:20
🔗
|
|
odemg has quit IRC (Remote host closed the connection) |
|
23:24
🔗
|
|
BlueMaxim has joined #archiveteam |
|
23:25
🔗
|
|
Asparagir has quit IRC (Asparagir) |
|
23:28
🔗
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
23:29
🔗
|
|
odemg has joined #archiveteam |
|
23:32
🔗
|
|
dashcloud has joined #archiveteam |
|
23:47
🔗
|
|
odemg has quit IRC (Remote host closed the connection) |
|
23:50
🔗
|
|
odemg has joined #archiveteam |
|
23:52
🔗
|
|
eightfold has joined #archiveteam |
|
23:59
🔗
|
|
BlueMaxim has quit IRC (Quit: Leaving) |