| Time |
Nickname |
Message |
|
00:03
🔗
|
|
mutoso has joined #archiveteam |
|
00:04
🔗
|
|
ky0ko has quit IRC (Read error: Operation timed out) |
|
00:07
🔗
|
|
ky0ko has joined #archiveteam |
|
00:12
🔗
|
|
AlexLehm has quit IRC (Ping timeout: 260 seconds) |
|
00:35
🔗
|
|
RichardG_ has joined #archiveteam |
|
00:36
🔗
|
|
RichardG has quit IRC (Read error: Operation timed out) |
|
00:42
🔗
|
|
JesseW has joined #archiveteam |
|
00:47
🔗
|
|
Honno has quit IRC (Read error: Operation timed out) |
|
00:54
🔗
|
|
brayden has joined #archiveteam |
|
00:54
🔗
|
|
swebb sets mode: +o brayden |
|
01:04
🔗
|
|
zenguy_pc has quit IRC (Read error: Operation timed out) |
|
01:05
🔗
|
|
BlueMaxim has joined #archiveteam |
|
01:19
🔗
|
|
zenguy_pc has joined #archiveteam |
|
01:19
🔗
|
|
kristian_ has quit IRC (Quit: Leaving) |
|
01:35
🔗
|
|
zenguy_pc has quit IRC (Read error: Operation timed out) |
|
01:52
🔗
|
|
zenguy_pc has joined #archiveteam |
|
01:56
🔗
|
|
zenguy_pc has quit IRC (Read error: Operation timed out) |
|
02:04
🔗
|
|
necron99 has joined #archiveteam |
|
02:05
🔗
|
necron99 |
if you haven't already heard, there is serious uncertainty surrounding the future of teksyndicate.com and its forums.teksyndicate.com |
|
02:05
🔗
|
|
necron99 has quit IRC () |
|
02:06
🔗
|
|
necron99 has joined #archiveteam |
|
02:12
🔗
|
wp494 |
wendell has the actual box that the forums are hosted on |
|
02:12
🔗
|
wp494 |
logan just has domain name |
|
02:12
🔗
|
wp494 |
(and cloudflare access, too) |
|
02:12
🔗
|
wp494 |
still worth having a look at and archivebotting when possible |
|
02:15
🔗
|
necron99 |
users can export their own posts too |
|
02:16
🔗
|
necron99 |
anyway thx for noticing |
|
02:28
🔗
|
|
necron99 has left |
|
02:34
🔗
|
|
zenguy_pc has joined #archiveteam |
|
02:37
🔗
|
|
RichardG_ has quit IRC (Ping timeout: 260 seconds) |
|
02:38
🔗
|
|
RichardG has joined #archiveteam |
|
02:41
🔗
|
|
RichardG_ has joined #archiveteam |
|
02:48
🔗
|
|
RichardG has quit IRC (Read error: Operation timed out) |
|
02:56
🔗
|
odie5533 |
Apparently the disappearance of the Firefall site/forum was due to server errors and it is back. Is there anyone that can confirm the entire forum was properly archived? |
|
02:56
🔗
|
odie5533 |
Someone in here had said the bot died while trying to archvie it. |
|
02:56
🔗
|
|
TC01_ is now known as TC01 |
|
03:00
🔗
|
|
zenguy_pc has quit IRC (Read error: Operation timed out) |
|
03:04
🔗
|
|
RichardG_ has quit IRC (Ping timeout: 255 seconds) |
|
03:31
🔗
|
|
zenguy_pc has joined #archiveteam |
|
03:36
🔗
|
|
zenguy_pc has quit IRC (Ping timeout: 244 seconds) |
|
04:08
🔗
|
|
Sk1d has quit IRC (Ping timeout: 194 seconds) |
|
04:11
🔗
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
04:14
🔗
|
|
Sk1d has joined #archiveteam |
|
04:38
🔗
|
|
Meroje has quit IRC (Quit: bye!) |
|
04:39
🔗
|
|
Meroje has joined #archiveteam |
|
05:17
🔗
|
|
ravetcofx has quit IRC (Ping timeout: 246 seconds) |
|
05:35
🔗
|
|
ravetcofx has joined #archiveteam |
|
06:08
🔗
|
|
JesseW has quit IRC (Ping timeout: 370 seconds) |
|
06:42
🔗
|
|
zenguy_pc has joined #archiveteam |
|
06:46
🔗
|
Medowar |
lol. APC have a password-protected ftp server(ftp.apc.com), but have hardcoded credentials with every downloadlink. |
|
06:46
🔗
|
Medowar |
ftp://restrict:Kop$74!@ftp.apc.com/ gives full access |
|
06:47
🔗
|
Medowar |
correction: full read access |
|
06:50
🔗
|
yipdw |
you can browse the FTP anonymously anyway |
|
06:53
🔗
|
Medowar |
yes, but you dont get access to the real files |
|
06:57
🔗
|
|
sivoais has quit IRC (Read error: Operation timed out) |
|
06:59
🔗
|
|
espes__ has quit IRC (Ping timeout: 250 seconds) |
|
07:08
🔗
|
|
sivoais has joined #archiveteam |
|
07:17
🔗
|
|
espes__ has joined #archiveteam |
|
07:23
🔗
|
|
espes__ has quit IRC (Ping timeout: 250 seconds) |
|
07:48
🔗
|
|
espes__ has joined #archiveteam |
|
07:51
🔗
|
|
schbirid has joined #archiveteam |
|
07:56
🔗
|
|
DrKyonko has joined #archiveteam |
|
08:28
🔗
|
Kksmkrn |
I'm backing off a bit on nujij, it's starting to throw 503s at me |
|
08:56
🔗
|
Jeroen__u |
Kksmkrn: I have started a lot of jobs on both my warriors and standalone, so that might be why. |
|
08:57
🔗
|
Jeroen__u |
Kksmkrn: However I think that vantec is the biggest fish in the pond. |
|
08:57
🔗
|
Jeroen__u |
Might be useful to lower the rate limit on nujij. |
|
08:58
🔗
|
Kksmkrn |
Jeroen__u: Just as long as it gets done, though bashing it like this.. |
|
08:58
🔗
|
Jeroen__u |
Kksmkrn: There is a lot to do, and it is shutting down relatively soon. |
|
08:59
🔗
|
Jeroen__u |
The current machines I am running will cost me about 12 cent per hour to run. |
|
09:02
🔗
|
Kksmkrn |
Jeroen__u: I get that, it being a big job. Though everything I contribute is currently done from home as I'm limited in funds so yea :) |
|
09:03
🔗
|
Jeroen__u |
Kksmkrn: I am still a student, so I am limited in funds as well. But luckily I have access to a lot of cheap hardware and fast network connections. |
|
09:03
🔗
|
|
MMovie2 has joined #archiveteam |
|
09:03
🔗
|
|
MMovie has quit IRC (Read error: Operation timed out) |
|
09:05
🔗
|
|
Honno has joined #archiveteam |
|
09:07
🔗
|
|
espes__ has quit IRC (Ping timeout: 250 seconds) |
|
09:07
🔗
|
Jeroen__u |
Oh shit, I am volunteering so much that the tracker once 503 for me. |
|
09:07
🔗
|
Jeroen__u |
Not nujij, but the tracker. |
|
09:08
🔗
|
Kksmkrn |
Heh |
|
09:09
🔗
|
Jeroen__u |
I have 23 VMs running that are trying to get their jobs from the tracker. |
|
09:10
🔗
|
Jeroen__u |
In total 348 concurrent tasks trying to get their jobs. |
|
09:13
🔗
|
|
espes__ has joined #archiveteam |
|
09:17
🔗
|
Jeroen__u |
Looks like nujij can finally handle all these warriors, maybe up the limit a bit? |
|
09:19
🔗
|
Igloo^ |
Jeroen__u: we keep the concurrency low to now DDOS the site |
|
09:19
🔗
|
Igloo^ |
:) |
|
09:19
🔗
|
Jeroen__u |
Igloo^: Maybe I can retire 10 VMs :^) |
|
09:20
🔗
|
Igloo^ |
lol, How much compute do you have? :p |
|
09:20
🔗
|
Igloo^ |
(I run the scrips manually on single VMs with > 80 processes as that's more efficient for my equipment) |
|
09:22
🔗
|
Jeroen__u |
I have like 15 Digital Ocean droplets with minimum size, each running 20 processes concurrently. I am also running 8 Warrior VMs on my PC each 6 processes concurrently. |
|
09:22
🔗
|
Jeroen__u |
How can you run 80 processes without it throwing an error, stating that running more than 20 bugs out? |
|
09:23
🔗
|
Igloo^ |
Screen :) |
|
09:23
🔗
|
Igloo^ |
Multiple screen instances |
|
09:23
🔗
|
Jeroen__u |
Whenever I do that it says that a socket is already getting used, I think. |
|
09:24
🔗
|
Jeroen__u |
socket.error: [Errno 98] Address already in use |
|
09:24
🔗
|
Jeroen__u |
Igloo^: Do you have multiple IP addresses? |
|
09:25
🔗
|
Jeroen__u |
Maybe if I give it multiple network interfaces it will work. |
|
09:25
🔗
|
Medowar |
Jeroen__u: you can use different ports with --port |
|
09:25
🔗
|
Jeroen__u |
Oh wait |
|
09:25
🔗
|
Medowar |
like --port 1331 |
|
09:26
🔗
|
Jeroen__u |
I see. |
|
09:26
🔗
|
Jeroen__u |
Thanks, |
|
09:26
🔗
|
Medowar |
also #archiveteam-bs |
|
09:30
🔗
|
yipdw |
Jeroen__u: nujij is set to a maximum of 70 requests per minute; 348 is obviously overkill |
|
09:30
🔗
|
yipdw |
please back off voluntarily |
|
09:30
🔗
|
Jeroen__u |
yipdw: I am backing off, I shut down 70 servers already. |
|
09:30
🔗
|
Jeroen__u |
10& |
|
09:30
🔗
|
Jeroen__u |
10* |
|
09:33
🔗
|
yipdw |
Jeroen__u: ok. you can probably back even more off; the request:grant ratio is currently 7.5:1 |
|
09:34
🔗
|
Jeroen__u |
yipdw: It should go down over the next few minutes, the VMs have been shut down recently. |
|
09:35
🔗
|
yipdw |
I was going to say if you need to have a gazillion instances running for some reason you could throw some of it at URLTeam |
|
09:35
🔗
|
Jeroen__u |
yipdw: I might do that, I hate URL shorteners. |
|
09:36
🔗
|
Jeroen__u |
So, who is the vantec guy? He is hogging a lot of jobs. |
|
09:36
🔗
|
yipdw |
not here right now |
|
09:40
🔗
|
yipdw |
job hogging isn't really a big problem |
|
09:40
🔗
|
yipdw |
vantec might be running a lot of jobs but their system has also not checked in ~9100 items |
|
09:40
🔗
|
yipdw |
it'll just get requeued and in time will be routed to a more modest, more reliable setup |
|
09:41
🔗
|
Jeroen__u |
Could you explain it a bit further? |
|
09:41
🔗
|
Jeroen__u |
I don't quite understand it. |
|
09:41
🔗
|
yipdw |
each work item covers a certain article range |
|
09:42
🔗
|
yipdw |
it can either succeed or fail; failed ones are detected by "hmm this hasn't reported back in a few days but we expect it to come back within an hour or two" |
|
09:43
🔗
|
yipdw |
those that fail are thrown back into the queue and eventually they'll succeed, either by getting onto a system that won't eat itself or via script updates to e.g. handle some anomalous situation that shows up in that work item |
|
09:43
🔗
|
Jeroen__u |
Do not worry, I have shut down my workers gracefully. |
|
09:44
🔗
|
Igloo^ |
Jeroen__u: you can just run the URLTeam scripts |
|
09:44
🔗
|
Igloo^ |
There are a couple of other projects coming up too |
|
09:45
🔗
|
Jeroen__u |
yipdw: how can I take many articles without making enemies here? |
|
09:45
🔗
|
Igloo^ |
You won't make enemies :) |
|
09:45
🔗
|
yipdw |
you can't and that's kind of the point |
|
09:45
🔗
|
Igloo^ |
We need the resources at the moment... The tracker prevents one person hogging the whole lot |
|
09:45
🔗
|
Jeroen__u |
Igloo^: I will run the URLTeam scripts too, but nujij is priority for me as I used to look at conversations over there. |
|
09:46
🔗
|
Igloo^ |
=] |
|
09:47
🔗
|
Jeroen__u |
yipdw: What is the current request:grant ratio? I have shut down many jobs. |
|
09:47
🔗
|
Jeroen__u |
I have 48 left. |
|
09:47
🔗
|
yipdw |
3:1 |
|
09:47
🔗
|
yipdw |
that's global |
|
09:53
🔗
|
Jeroen__u |
Global meaning over all projects? |
|
09:53
🔗
|
yipdw |
sorry, global meaning across all requests on nujij |
|
09:54
🔗
|
|
DrKyonko has quit IRC (Quit: Depression is merely anger without enthusiasm) |
|
09:57
🔗
|
|
BlueMaxim has quit IRC (Quit: Leaving) |
|
10:19
🔗
|
|
AlexLehm has joined #archiveteam |
|
11:09
🔗
|
arkiver |
Jeroen__u: you can run yahooanswers. |
|
11:09
🔗
|
arkiver |
We need a lot of IPs on that project, the more IPs the better |
|
11:09
🔗
|
arkiver |
2 concurrent per IP is adviced |
|
11:09
🔗
|
Jeroen__u |
I read about that. |
|
11:18
🔗
|
|
irl has joined #archiveteam |
|
11:18
🔗
|
|
zenguy_pc has quit IRC (Ping timeout: 260 seconds) |
|
11:53
🔗
|
|
dashcloud has joined #archiveteam |
|
12:24
🔗
|
|
tuankiet has quit IRC (Ping timeout: 244 seconds) |
|
12:24
🔗
|
|
tuankiet has joined #archiveteam |
|
12:53
🔗
|
|
ats has quit IRC (Read error: Operation timed out) |
|
13:06
🔗
|
|
Whopper has quit IRC (Ping timeout: 370 seconds) |
|
13:12
🔗
|
|
ats has joined #archiveteam |
|
13:12
🔗
|
|
brayden has quit IRC (Read error: Operation timed out) |
|
13:13
🔗
|
|
ky0ko has quit IRC (Read error: Operation timed out) |
|
13:21
🔗
|
|
ky0ko has joined #archiveteam |
|
13:26
🔗
|
|
brayden has joined #archiveteam |
|
13:26
🔗
|
|
swebb sets mode: +o brayden |
|
13:36
🔗
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
13:41
🔗
|
luckcolor |
arkiver: on it. i hope the bug has been fixed |
|
13:41
🔗
|
arkiver |
err, no |
|
13:41
🔗
|
arkiver |
sorry, forgot to take a look |
|
13:42
🔗
|
luckcolor |
ah np |
|
13:42
🔗
|
arkiver |
will do now |
|
13:42
🔗
|
luckcolor |
you should still have it in your query |
|
13:42
🔗
|
luckcolor |
thanks |
|
13:42
🔗
|
arkiver |
yeah, got it in my logs |
|
13:46
🔗
|
arkiver |
fixed. |
|
13:53
🔗
|
|
ravetcofx has quit IRC (Ping timeout: 370 seconds) |
|
14:00
🔗
|
Jeroen__u |
I know that I'm new here, but I suggest that the nujij maximum requests per minute to be raised from 70 to 100 during 1 AM Europe/Amsterdam time to 6 AM Europe/Amsterdam time. |
|
14:43
🔗
|
|
WinterFox has quit IRC (Read error: Operation timed out) |
|
14:47
🔗
|
|
atomotic has joined #archiveteam |
|
15:34
🔗
|
|
ravetcofx has joined #archiveteam |
|
15:47
🔗
|
|
ravetcofx has quit IRC (Read error: Operation timed out) |
|
15:49
🔗
|
|
atomotic has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…) |
|
15:52
🔗
|
|
ravetcofx has joined #archiveteam |
|
16:02
🔗
|
|
JesseW has joined #archiveteam |
|
16:11
🔗
|
|
metal_cam has joined #archiveteam |
|
16:14
🔗
|
|
BartoCH has quit IRC (Ping timeout: 260 seconds) |
|
16:14
🔗
|
|
kristian_ has joined #archiveteam |
|
16:31
🔗
|
|
JesseW has quit IRC (Ping timeout: 370 seconds) |
|
16:39
🔗
|
|
atomotic has joined #archiveteam |
|
16:47
🔗
|
|
JW_work1 has joined #archiveteam |
|
16:48
🔗
|
|
BartoCH has joined #archiveteam |
|
16:49
🔗
|
|
JW_work has quit IRC (Read error: Operation timed out) |
|
16:51
🔗
|
|
JW_work1 has quit IRC (Client Quit) |
|
17:13
🔗
|
|
RichardG has joined #archiveteam |
|
17:13
🔗
|
|
RichardG has quit IRC (Client Quit) |
|
17:13
🔗
|
|
RichardG has joined #archiveteam |
|
17:34
🔗
|
|
Simpbrain has quit IRC (Ping timeout: 370 seconds) |
|
17:34
🔗
|
|
Simpbrain has joined #archiveteam |
|
17:37
🔗
|
|
bRick5772 has joined #archiveteam |
|
17:53
🔗
|
|
Morbus has quit IRC (Read error: Operation timed out) |
|
17:58
🔗
|
|
jeroen52_ has joined #archiveteam |
|
17:59
🔗
|
|
Jeroen__u has quit IRC (Ping timeout: 268 seconds) |
|
18:01
🔗
|
|
ravetcofx has quit IRC (Read error: Operation timed out) |
|
18:04
🔗
|
|
RichardG_ has joined #archiveteam |
|
18:04
🔗
|
|
RichardG has quit IRC (Read error: Operation timed out) |
|
18:09
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
|
18:19
🔗
|
|
RichardG_ has quit IRC (Ping timeout: 370 seconds) |
|
18:23
🔗
|
|
tuankiet has quit IRC (Ping timeout: 244 seconds) |
|
18:36
🔗
|
|
tuankiet has joined #archiveteam |
|
18:54
🔗
|
|
JW_work has joined #archiveteam |
|
19:19
🔗
|
|
metalcamp has joined #archiveteam |
|
19:22
🔗
|
|
metal_cam has quit IRC (Read error: Operation timed out) |
|
19:29
🔗
|
|
JW_work1 has joined #archiveteam |
|
19:31
🔗
|
|
JW_work has quit IRC (Read error: Operation timed out) |
|
19:37
🔗
|
|
JW_work1 has quit IRC (Read error: Operation timed out) |
|
19:42
🔗
|
|
ndiddy has joined #archiveteam |
|
19:43
🔗
|
|
maseck has quit IRC (Remote host closed the connection) |
|
19:51
🔗
|
|
maseck has joined #archiveteam |
|
20:16
🔗
|
|
metalcamp has quit IRC (Read error: Operation timed out) |
|
20:16
🔗
|
|
BartoCH has quit IRC (Ping timeout: 260 seconds) |
|
20:18
🔗
|
|
BartoCH has joined #archiveteam |
|
20:32
🔗
|
|
Jeroen52 has quit IRC (Read error: Connection reset by peer) |
|
20:39
🔗
|
bRick5772 |
has anyone successfully tried to run warrior-dockerfile on a raspberry pi? just tried with latest hypriot image but container fails to start... |
|
20:43
🔗
|
|
dashcloud has joined #archiveteam |
|
20:44
🔗
|
BartoCH |
bRick5772: i tried, docker is only x86_64 |
|
20:45
🔗
|
bRick5772 |
yes, it used to be - but docker can be build on arm (since docker 1.10) |
|
20:46
🔗
|
bRick5772 |
and that's what the hypriot guys did - build docker for arm based on raspbian and release a docker-prepared image: http://blog.hypriot.com/getting-started-with-docker-on-your-arm-device/ |
|
20:48
🔗
|
BartoCH |
mhh, so you'd need your own dockerfile then, is it? I guess the current one has some specific x86_64 binaries in there. |
|
20:48
🔗
|
bRick5772 |
but i guess the stuff that's being loaded in the warrior-dockerfile is not arm-compatible - just wondering if someone succeeded with a raspi running the warrior-dockerfile |
|
20:48
🔗
|
bRick5772 |
ok |
|
20:48
🔗
|
BartoCH |
you'd better run yourself the grab script in a daemonized screen session |
|
20:49
🔗
|
BartoCH |
those are written in python afaik, it should work |
|
20:49
🔗
|
bRick5772 |
the logfile says something about"fatal error: rt_sigaction failure" when calling some .go files? |
|
20:51
🔗
|
bRick5772 |
hmm, ok - well I like the idea of having a raspi sitting around just being dedicated to archiving-tasks and being supplied with jobs automatically |
|
21:10
🔗
|
|
Jeroen52 has joined #archiveteam |
|
21:11
🔗
|
|
ky0ko has quit IRC (Ping timeout: 244 seconds) |
|
21:12
🔗
|
|
arkiver2 has joined #archiveteam |
|
21:21
🔗
|
Kaz |
bRick5772: if you're up for the work, there's nothing stopping you from actually using the warrior scripts, I don't hink |
|
21:21
🔗
|
Kaz |
it's pretty much python all the way down, iirc |
|
21:22
🔗
|
Kaz |
http://archiveteam.org/index.php?title=Dev/Warrior#Bootup |
|
21:32
🔗
|
|
schbirid has quit IRC (Quit: Leaving) |
|
21:33
🔗
|
bRick5772 |
Kaz, thx I will give it a go - just tried to simply run boot.sh on the pi but there were some errors as well - I'll try to get it running and will let you know if there's something to improve |
|
21:34
🔗
|
bRick5772 |
after all I think raspis are great hardware for such tasks... |
|
21:36
🔗
|
BartoCH |
raspis are not really that great for heavy IO stuffs, at least the one i have which is kinda becoming old |
|
21:36
🔗
|
BartoCH |
USB and ethernet are using a shared bus, this can really become problematic |
|
21:38
🔗
|
BartoCH |
but i'd be interested to hear if I'm wrong, my pi is really old now and they may have mitigated the problem |
|
21:39
🔗
|
Kaz |
-bs alarm |
|
21:39
🔗
|
|
ky0ko has joined #archiveteam |
|
21:43
🔗
|
|
RichardG has joined #archiveteam |
|
21:49
🔗
|
bRick5772 |
BartoCH: afaik the newer raspis (i got a raspi2) still share ethernet and usb on the same bus - but considering me being online via cable (30mbit) I guess the raspi will be able to handle the data. anyway, I'll let you know how it goes. |
|
21:59
🔗
|
|
BlueMaxim has joined #archiveteam |
|
22:11
🔗
|
|
bRick5772 has quit IRC (Quit: Leaving.) |
|
22:14
🔗
|
|
RichardG has quit IRC (Read error: Operation timed out) |
|
22:18
🔗
|
|
bsmith093 has joined #archiveteam |
|
22:24
🔗
|
|
Honno has quit IRC (Read error: Operation timed out) |
|
22:52
🔗
|
|
maelstrom has joined #archiveteam |
|
23:05
🔗
|
|
WinterFox has joined #archiveteam |
|
23:10
🔗
|
|
jdude104 has quit IRC (Ping timeout: 268 seconds) |
|
23:55
🔗
|
|
melas has joined #archiveteam |