#archiveteam 2016-08-31,Wed

↑back Search

Time Nickname Message
00:03 🔗 mutoso has joined #archiveteam
00:04 🔗 ky0ko has quit IRC (Read error: Operation timed out)
00:07 🔗 ky0ko has joined #archiveteam
00:12 🔗 AlexLehm has quit IRC (Ping timeout: 260 seconds)
00:35 🔗 RichardG_ has joined #archiveteam
00:36 🔗 RichardG has quit IRC (Read error: Operation timed out)
00:42 🔗 JesseW has joined #archiveteam
00:47 🔗 Honno has quit IRC (Read error: Operation timed out)
00:54 🔗 brayden has joined #archiveteam
00:54 🔗 swebb sets mode: +o brayden
01:04 🔗 zenguy_pc has quit IRC (Read error: Operation timed out)
01:05 🔗 BlueMaxim has joined #archiveteam
01:19 🔗 zenguy_pc has joined #archiveteam
01:19 🔗 kristian_ has quit IRC (Quit: Leaving)
01:35 🔗 zenguy_pc has quit IRC (Read error: Operation timed out)
01:52 🔗 zenguy_pc has joined #archiveteam
01:56 🔗 zenguy_pc has quit IRC (Read error: Operation timed out)
02:04 🔗 necron99 has joined #archiveteam
02:05 🔗 necron99 if you haven't already heard, there is serious uncertainty surrounding the future of teksyndicate.com and its forums.teksyndicate.com
02:05 🔗 necron99 has quit IRC ()
02:06 🔗 necron99 has joined #archiveteam
02:12 🔗 wp494 wendell has the actual box that the forums are hosted on
02:12 🔗 wp494 logan just has domain name
02:12 🔗 wp494 (and cloudflare access, too)
02:12 🔗 wp494 still worth having a look at and archivebotting when possible
02:15 🔗 necron99 users can export their own posts too
02:16 🔗 necron99 anyway thx for noticing
02:28 🔗 necron99 has left
02:34 🔗 zenguy_pc has joined #archiveteam
02:37 🔗 RichardG_ has quit IRC (Ping timeout: 260 seconds)
02:38 🔗 RichardG has joined #archiveteam
02:41 🔗 RichardG_ has joined #archiveteam
02:48 🔗 RichardG has quit IRC (Read error: Operation timed out)
02:56 🔗 odie5533 Apparently the disappearance of the Firefall site/forum was due to server errors and it is back. Is there anyone that can confirm the entire forum was properly archived?
02:56 🔗 odie5533 Someone in here had said the bot died while trying to archvie it.
02:56 🔗 TC01_ is now known as TC01
03:00 🔗 zenguy_pc has quit IRC (Read error: Operation timed out)
03:04 🔗 RichardG_ has quit IRC (Ping timeout: 255 seconds)
03:31 🔗 zenguy_pc has joined #archiveteam
03:36 🔗 zenguy_pc has quit IRC (Ping timeout: 244 seconds)
04:08 🔗 Sk1d has quit IRC (Ping timeout: 194 seconds)
04:11 🔗 dashcloud has quit IRC (Read error: Operation timed out)
04:14 🔗 Sk1d has joined #archiveteam
04:38 🔗 Meroje has quit IRC (Quit: bye!)
04:39 🔗 Meroje has joined #archiveteam
05:17 🔗 ravetcofx has quit IRC (Ping timeout: 246 seconds)
05:35 🔗 ravetcofx has joined #archiveteam
06:08 🔗 JesseW has quit IRC (Ping timeout: 370 seconds)
06:42 🔗 zenguy_pc has joined #archiveteam
06:46 🔗 Medowar lol. APC have a password-protected ftp server(ftp.apc.com), but have hardcoded credentials with every downloadlink.
06:46 🔗 Medowar ftp://restrict:Kop$74!@ftp.apc.com/ gives full access
06:47 🔗 Medowar correction: full read access
06:50 🔗 yipdw you can browse the FTP anonymously anyway
06:53 🔗 Medowar yes, but you dont get access to the real files
06:57 🔗 sivoais has quit IRC (Read error: Operation timed out)
06:59 🔗 espes__ has quit IRC (Ping timeout: 250 seconds)
07:08 🔗 sivoais has joined #archiveteam
07:17 🔗 espes__ has joined #archiveteam
07:23 🔗 espes__ has quit IRC (Ping timeout: 250 seconds)
07:48 🔗 espes__ has joined #archiveteam
07:51 🔗 schbirid has joined #archiveteam
07:56 🔗 DrKyonko has joined #archiveteam
08:28 🔗 Kksmkrn I'm backing off a bit on nujij, it's starting to throw 503s at me
08:56 🔗 Jeroen__u Kksmkrn: I have started a lot of jobs on both my warriors and standalone, so that might be why.
08:57 🔗 Jeroen__u Kksmkrn: However I think that vantec is the biggest fish in the pond.
08:57 🔗 Jeroen__u Might be useful to lower the rate limit on nujij.
08:58 🔗 Kksmkrn Jeroen__u: Just as long as it gets done, though bashing it like this..
08:58 🔗 Jeroen__u Kksmkrn: There is a lot to do, and it is shutting down relatively soon.
08:59 🔗 Jeroen__u The current machines I am running will cost me about 12 cent per hour to run.
09:02 🔗 Kksmkrn Jeroen__u: I get that, it being a big job. Though everything I contribute is currently done from home as I'm limited in funds so yea :)
09:03 🔗 Jeroen__u Kksmkrn: I am still a student, so I am limited in funds as well. But luckily I have access to a lot of cheap hardware and fast network connections.
09:03 🔗 MMovie2 has joined #archiveteam
09:03 🔗 MMovie has quit IRC (Read error: Operation timed out)
09:05 🔗 Honno has joined #archiveteam
09:07 🔗 espes__ has quit IRC (Ping timeout: 250 seconds)
09:07 🔗 Jeroen__u Oh shit, I am volunteering so much that the tracker once 503 for me.
09:07 🔗 Jeroen__u Not nujij, but the tracker.
09:08 🔗 Kksmkrn Heh
09:09 🔗 Jeroen__u I have 23 VMs running that are trying to get their jobs from the tracker.
09:10 🔗 Jeroen__u In total 348 concurrent tasks trying to get their jobs.
09:13 🔗 espes__ has joined #archiveteam
09:17 🔗 Jeroen__u Looks like nujij can finally handle all these warriors, maybe up the limit a bit?
09:19 🔗 Igloo^ Jeroen__u: we keep the concurrency low to now DDOS the site
09:19 🔗 Igloo^ :)
09:19 🔗 Jeroen__u Igloo^: Maybe I can retire 10 VMs :^)
09:20 🔗 Igloo^ lol, How much compute do you have? :p
09:20 🔗 Igloo^ (I run the scrips manually on single VMs with > 80 processes as that's more efficient for my equipment)
09:22 🔗 Jeroen__u I have like 15 Digital Ocean droplets with minimum size, each running 20 processes concurrently. I am also running 8 Warrior VMs on my PC each 6 processes concurrently.
09:22 🔗 Jeroen__u How can you run 80 processes without it throwing an error, stating that running more than 20 bugs out?
09:23 🔗 Igloo^ Screen :)
09:23 🔗 Igloo^ Multiple screen instances
09:23 🔗 Jeroen__u Whenever I do that it says that a socket is already getting used, I think.
09:24 🔗 Jeroen__u socket.error: [Errno 98] Address already in use
09:24 🔗 Jeroen__u Igloo^: Do you have multiple IP addresses?
09:25 🔗 Jeroen__u Maybe if I give it multiple network interfaces it will work.
09:25 🔗 Medowar Jeroen__u: you can use different ports with --port
09:25 🔗 Jeroen__u Oh wait
09:25 🔗 Medowar like --port 1331
09:26 🔗 Jeroen__u I see.
09:26 🔗 Jeroen__u Thanks,
09:26 🔗 Medowar also #archiveteam-bs
09:30 🔗 yipdw Jeroen__u: nujij is set to a maximum of 70 requests per minute; 348 is obviously overkill
09:30 🔗 yipdw please back off voluntarily
09:30 🔗 Jeroen__u yipdw: I am backing off, I shut down 70 servers already.
09:30 🔗 Jeroen__u 10&
09:30 🔗 Jeroen__u 10*
09:33 🔗 yipdw Jeroen__u: ok. you can probably back even more off; the request:grant ratio is currently 7.5:1
09:34 🔗 Jeroen__u yipdw: It should go down over the next few minutes, the VMs have been shut down recently.
09:35 🔗 yipdw I was going to say if you need to have a gazillion instances running for some reason you could throw some of it at URLTeam
09:35 🔗 Jeroen__u yipdw: I might do that, I hate URL shorteners.
09:36 🔗 Jeroen__u So, who is the vantec guy? He is hogging a lot of jobs.
09:36 🔗 yipdw not here right now
09:40 🔗 yipdw job hogging isn't really a big problem
09:40 🔗 yipdw vantec might be running a lot of jobs but their system has also not checked in ~9100 items
09:40 🔗 yipdw it'll just get requeued and in time will be routed to a more modest, more reliable setup
09:41 🔗 Jeroen__u Could you explain it a bit further?
09:41 🔗 Jeroen__u I don't quite understand it.
09:41 🔗 yipdw each work item covers a certain article range
09:42 🔗 yipdw it can either succeed or fail; failed ones are detected by "hmm this hasn't reported back in a few days but we expect it to come back within an hour or two"
09:43 🔗 yipdw those that fail are thrown back into the queue and eventually they'll succeed, either by getting onto a system that won't eat itself or via script updates to e.g. handle some anomalous situation that shows up in that work item
09:43 🔗 Jeroen__u Do not worry, I have shut down my workers gracefully.
09:44 🔗 Igloo^ Jeroen__u: you can just run the URLTeam scripts
09:44 🔗 Igloo^ There are a couple of other projects coming up too
09:45 🔗 Jeroen__u yipdw: how can I take many articles without making enemies here?
09:45 🔗 Igloo^ You won't make enemies :)
09:45 🔗 yipdw you can't and that's kind of the point
09:45 🔗 Igloo^ We need the resources at the moment... The tracker prevents one person hogging the whole lot
09:45 🔗 Jeroen__u Igloo^: I will run the URLTeam scripts too, but nujij is priority for me as I used to look at conversations over there.
09:46 🔗 Igloo^ =]
09:47 🔗 Jeroen__u yipdw: What is the current request:grant ratio? I have shut down many jobs.
09:47 🔗 Jeroen__u I have 48 left.
09:47 🔗 yipdw 3:1
09:47 🔗 yipdw that's global
09:53 🔗 Jeroen__u Global meaning over all projects?
09:53 🔗 yipdw sorry, global meaning across all requests on nujij
09:54 🔗 DrKyonko has quit IRC (Quit: Depression is merely anger without enthusiasm)
09:57 🔗 BlueMaxim has quit IRC (Quit: Leaving)
10:19 🔗 AlexLehm has joined #archiveteam
11:09 🔗 arkiver Jeroen__u: you can run yahooanswers.
11:09 🔗 arkiver We need a lot of IPs on that project, the more IPs the better
11:09 🔗 arkiver 2 concurrent per IP is adviced
11:09 🔗 Jeroen__u I read about that.
11:18 🔗 irl has joined #archiveteam
11:18 🔗 zenguy_pc has quit IRC (Ping timeout: 260 seconds)
11:53 🔗 dashcloud has joined #archiveteam
12:24 🔗 tuankiet has quit IRC (Ping timeout: 244 seconds)
12:24 🔗 tuankiet has joined #archiveteam
12:53 🔗 ats has quit IRC (Read error: Operation timed out)
13:06 🔗 Whopper has quit IRC (Ping timeout: 370 seconds)
13:12 🔗 ats has joined #archiveteam
13:12 🔗 brayden has quit IRC (Read error: Operation timed out)
13:13 🔗 ky0ko has quit IRC (Read error: Operation timed out)
13:21 🔗 ky0ko has joined #archiveteam
13:26 🔗 brayden has joined #archiveteam
13:26 🔗 swebb sets mode: +o brayden
13:36 🔗 dashcloud has quit IRC (Read error: Operation timed out)
13:41 🔗 luckcolor arkiver: on it. i hope the bug has been fixed
13:41 🔗 arkiver err, no
13:41 🔗 arkiver sorry, forgot to take a look
13:42 🔗 luckcolor ah np
13:42 🔗 arkiver will do now
13:42 🔗 luckcolor you should still have it in your query
13:42 🔗 luckcolor thanks
13:42 🔗 arkiver yeah, got it in my logs
13:46 🔗 arkiver fixed.
13:53 🔗 ravetcofx has quit IRC (Ping timeout: 370 seconds)
14:00 🔗 Jeroen__u I know that I'm new here, but I suggest that the nujij maximum requests per minute to be raised from 70 to 100 during 1 AM Europe/Amsterdam time to 6 AM Europe/Amsterdam time.
14:43 🔗 WinterFox has quit IRC (Read error: Operation timed out)
14:47 🔗 atomotic has joined #archiveteam
15:34 🔗 ravetcofx has joined #archiveteam
15:47 🔗 ravetcofx has quit IRC (Read error: Operation timed out)
15:49 🔗 atomotic has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…)
15:52 🔗 ravetcofx has joined #archiveteam
16:02 🔗 JesseW has joined #archiveteam
16:11 🔗 metal_cam has joined #archiveteam
16:14 🔗 BartoCH has quit IRC (Ping timeout: 260 seconds)
16:14 🔗 kristian_ has joined #archiveteam
16:31 🔗 JesseW has quit IRC (Ping timeout: 370 seconds)
16:39 🔗 atomotic has joined #archiveteam
16:47 🔗 JW_work1 has joined #archiveteam
16:48 🔗 BartoCH has joined #archiveteam
16:49 🔗 JW_work has quit IRC (Read error: Operation timed out)
16:51 🔗 JW_work1 has quit IRC (Client Quit)
17:13 🔗 RichardG has joined #archiveteam
17:13 🔗 RichardG has quit IRC (Client Quit)
17:13 🔗 RichardG has joined #archiveteam
17:34 🔗 Simpbrain has quit IRC (Ping timeout: 370 seconds)
17:34 🔗 Simpbrain has joined #archiveteam
17:37 🔗 bRick5772 has joined #archiveteam
17:53 🔗 Morbus has quit IRC (Read error: Operation timed out)
17:58 🔗 jeroen52_ has joined #archiveteam
17:59 🔗 Jeroen__u has quit IRC (Ping timeout: 268 seconds)
18:01 🔗 ravetcofx has quit IRC (Read error: Operation timed out)
18:04 🔗 RichardG_ has joined #archiveteam
18:04 🔗 RichardG has quit IRC (Read error: Operation timed out)
18:09 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
18:19 🔗 RichardG_ has quit IRC (Ping timeout: 370 seconds)
18:23 🔗 tuankiet has quit IRC (Ping timeout: 244 seconds)
18:36 🔗 tuankiet has joined #archiveteam
18:54 🔗 JW_work has joined #archiveteam
19:19 🔗 metalcamp has joined #archiveteam
19:22 🔗 metal_cam has quit IRC (Read error: Operation timed out)
19:29 🔗 JW_work1 has joined #archiveteam
19:31 🔗 JW_work has quit IRC (Read error: Operation timed out)
19:37 🔗 JW_work1 has quit IRC (Read error: Operation timed out)
19:42 🔗 ndiddy has joined #archiveteam
19:43 🔗 maseck has quit IRC (Remote host closed the connection)
19:51 🔗 maseck has joined #archiveteam
20:16 🔗 metalcamp has quit IRC (Read error: Operation timed out)
20:16 🔗 BartoCH has quit IRC (Ping timeout: 260 seconds)
20:18 🔗 BartoCH has joined #archiveteam
20:32 🔗 Jeroen52 has quit IRC (Read error: Connection reset by peer)
20:39 🔗 bRick5772 has anyone successfully tried to run warrior-dockerfile on a raspberry pi? just tried with latest hypriot image but container fails to start...
20:43 🔗 dashcloud has joined #archiveteam
20:44 🔗 BartoCH bRick5772: i tried, docker is only x86_64
20:45 🔗 bRick5772 yes, it used to be - but docker can be build on arm (since docker 1.10)
20:46 🔗 bRick5772 and that's what the hypriot guys did - build docker for arm based on raspbian and release a docker-prepared image: http://blog.hypriot.com/getting-started-with-docker-on-your-arm-device/
20:48 🔗 BartoCH mhh, so you'd need your own dockerfile then, is it? I guess the current one has some specific x86_64 binaries in there.
20:48 🔗 bRick5772 but i guess the stuff that's being loaded in the warrior-dockerfile is not arm-compatible - just wondering if someone succeeded with a raspi running the warrior-dockerfile
20:48 🔗 bRick5772 ok
20:48 🔗 BartoCH you'd better run yourself the grab script in a daemonized screen session
20:49 🔗 BartoCH those are written in python afaik, it should work
20:49 🔗 bRick5772 the logfile says something about"fatal error: rt_sigaction failure" when calling some .go files?
20:51 🔗 bRick5772 hmm, ok - well I like the idea of having a raspi sitting around just being dedicated to archiving-tasks and being supplied with jobs automatically
21:10 🔗 Jeroen52 has joined #archiveteam
21:11 🔗 ky0ko has quit IRC (Ping timeout: 244 seconds)
21:12 🔗 arkiver2 has joined #archiveteam
21:21 🔗 Kaz bRick5772: if you're up for the work, there's nothing stopping you from actually using the warrior scripts, I don't hink
21:21 🔗 Kaz it's pretty much python all the way down, iirc
21:22 🔗 Kaz http://archiveteam.org/index.php?title=Dev/Warrior#Bootup
21:32 🔗 schbirid has quit IRC (Quit: Leaving)
21:33 🔗 bRick5772 Kaz, thx I will give it a go - just tried to simply run boot.sh on the pi but there were some errors as well - I'll try to get it running and will let you know if there's something to improve
21:34 🔗 bRick5772 after all I think raspis are great hardware for such tasks...
21:36 🔗 BartoCH raspis are not really that great for heavy IO stuffs, at least the one i have which is kinda becoming old
21:36 🔗 BartoCH USB and ethernet are using a shared bus, this can really become problematic
21:38 🔗 BartoCH but i'd be interested to hear if I'm wrong, my pi is really old now and they may have mitigated the problem
21:39 🔗 Kaz -bs alarm
21:39 🔗 ky0ko has joined #archiveteam
21:43 🔗 RichardG has joined #archiveteam
21:49 🔗 bRick5772 BartoCH: afaik the newer raspis (i got a raspi2) still share ethernet and usb on the same bus - but considering me being online via cable (30mbit) I guess the raspi will be able to handle the data. anyway, I'll let you know how it goes.
21:59 🔗 BlueMaxim has joined #archiveteam
22:11 🔗 bRick5772 has quit IRC (Quit: Leaving.)
22:14 🔗 RichardG has quit IRC (Read error: Operation timed out)
22:18 🔗 bsmith093 has joined #archiveteam
22:24 🔗 Honno has quit IRC (Read error: Operation timed out)
22:52 🔗 maelstrom has joined #archiveteam
23:05 🔗 WinterFox has joined #archiveteam
23:10 🔗 jdude104 has quit IRC (Ping timeout: 268 seconds)
23:55 🔗 melas has joined #archiveteam

irclogger-viewer