[00:03] *** mutoso has joined #archiveteam [00:04] *** ky0ko has quit IRC (Read error: Operation timed out) [00:07] *** ky0ko has joined #archiveteam [00:12] *** AlexLehm has quit IRC (Ping timeout: 260 seconds) [00:35] *** RichardG_ has joined #archiveteam [00:36] *** RichardG has quit IRC (Read error: Operation timed out) [00:42] *** JesseW has joined #archiveteam [00:47] *** Honno has quit IRC (Read error: Operation timed out) [00:54] *** brayden has joined #archiveteam [00:54] *** swebb sets mode: +o brayden [01:04] *** zenguy_pc has quit IRC (Read error: Operation timed out) [01:05] *** BlueMaxim has joined #archiveteam [01:19] *** zenguy_pc has joined #archiveteam [01:19] *** kristian_ has quit IRC (Quit: Leaving) [01:35] *** zenguy_pc has quit IRC (Read error: Operation timed out) [01:52] *** zenguy_pc has joined #archiveteam [01:56] *** zenguy_pc has quit IRC (Read error: Operation timed out) [02:04] *** necron99 has joined #archiveteam [02:05] if you haven't already heard, there is serious uncertainty surrounding the future of teksyndicate.com and its forums.teksyndicate.com [02:05] *** necron99 has quit IRC () [02:06] *** necron99 has joined #archiveteam [02:12] wendell has the actual box that the forums are hosted on [02:12] logan just has domain name [02:12] (and cloudflare access, too) [02:12] still worth having a look at and archivebotting when possible [02:15] users can export their own posts too [02:16] anyway thx for noticing [02:28] *** necron99 has left [02:34] *** zenguy_pc has joined #archiveteam [02:37] *** RichardG_ has quit IRC (Ping timeout: 260 seconds) [02:38] *** RichardG has joined #archiveteam [02:41] *** RichardG_ has joined #archiveteam [02:48] *** RichardG has quit IRC (Read error: Operation timed out) [02:56] Apparently the disappearance of the Firefall site/forum was due to server errors and it is back. Is there anyone that can confirm the entire forum was properly archived? [02:56] Someone in here had said the bot died while trying to archvie it. [02:56] *** TC01_ is now known as TC01 [03:00] *** zenguy_pc has quit IRC (Read error: Operation timed out) [03:04] *** RichardG_ has quit IRC (Ping timeout: 255 seconds) [03:31] *** zenguy_pc has joined #archiveteam [03:36] *** zenguy_pc has quit IRC (Ping timeout: 244 seconds) [04:08] *** Sk1d has quit IRC (Ping timeout: 194 seconds) [04:11] *** dashcloud has quit IRC (Read error: Operation timed out) [04:14] *** Sk1d has joined #archiveteam [04:38] *** Meroje has quit IRC (Quit: bye!) [04:39] *** Meroje has joined #archiveteam [05:17] *** ravetcofx has quit IRC (Ping timeout: 246 seconds) [05:35] *** ravetcofx has joined #archiveteam [06:08] *** JesseW has quit IRC (Ping timeout: 370 seconds) [06:42] *** zenguy_pc has joined #archiveteam [06:46] lol. APC have a password-protected ftp server(ftp.apc.com), but have hardcoded credentials with every downloadlink. [06:46] ftp://restrict:Kop$74!@ftp.apc.com/ gives full access [06:47] correction: full read access [06:50] you can browse the FTP anonymously anyway [06:53] yes, but you dont get access to the real files [06:57] *** sivoais has quit IRC (Read error: Operation timed out) [06:59] *** espes__ has quit IRC (Ping timeout: 250 seconds) [07:08] *** sivoais has joined #archiveteam [07:17] *** espes__ has joined #archiveteam [07:23] *** espes__ has quit IRC (Ping timeout: 250 seconds) [07:48] *** espes__ has joined #archiveteam [07:51] *** schbirid has joined #archiveteam [07:56] *** DrKyonko has joined #archiveteam [08:28] I'm backing off a bit on nujij, it's starting to throw 503s at me [08:56] Kksmkrn: I have started a lot of jobs on both my warriors and standalone, so that might be why. [08:57] Kksmkrn: However I think that vantec is the biggest fish in the pond. [08:57] Might be useful to lower the rate limit on nujij. [08:58] Jeroen__u: Just as long as it gets done, though bashing it like this.. [08:58] Kksmkrn: There is a lot to do, and it is shutting down relatively soon. [08:59] The current machines I am running will cost me about 12 cent per hour to run. [09:02] Jeroen__u: I get that, it being a big job. Though everything I contribute is currently done from home as I'm limited in funds so yea :) [09:03] Kksmkrn: I am still a student, so I am limited in funds as well. But luckily I have access to a lot of cheap hardware and fast network connections. [09:03] *** MMovie2 has joined #archiveteam [09:03] *** MMovie has quit IRC (Read error: Operation timed out) [09:05] *** Honno has joined #archiveteam [09:07] *** espes__ has quit IRC (Ping timeout: 250 seconds) [09:07] Oh shit, I am volunteering so much that the tracker once 503 for me. [09:07] Not nujij, but the tracker. [09:08] Heh [09:09] I have 23 VMs running that are trying to get their jobs from the tracker. [09:10] In total 348 concurrent tasks trying to get their jobs. [09:13] *** espes__ has joined #archiveteam [09:17] Looks like nujij can finally handle all these warriors, maybe up the limit a bit? [09:19] Jeroen__u: we keep the concurrency low to now DDOS the site [09:19] :) [09:19] Igloo^: Maybe I can retire 10 VMs :^) [09:20] lol, How much compute do you have? :p [09:20] (I run the scrips manually on single VMs with > 80 processes as that's more efficient for my equipment) [09:22] I have like 15 Digital Ocean droplets with minimum size, each running 20 processes concurrently. I am also running 8 Warrior VMs on my PC each 6 processes concurrently. [09:22] How can you run 80 processes without it throwing an error, stating that running more than 20 bugs out? [09:23] Screen :) [09:23] Multiple screen instances [09:23] Whenever I do that it says that a socket is already getting used, I think. [09:24] socket.error: [Errno 98] Address already in use [09:24] Igloo^: Do you have multiple IP addresses? [09:25] Maybe if I give it multiple network interfaces it will work. [09:25] Jeroen__u: you can use different ports with --port [09:25] Oh wait [09:25] like --port 1331 [09:26] I see. [09:26] Thanks, [09:26] also #archiveteam-bs [09:30] Jeroen__u: nujij is set to a maximum of 70 requests per minute; 348 is obviously overkill [09:30] please back off voluntarily [09:30] yipdw: I am backing off, I shut down 70 servers already. [09:30] 10& [09:30] 10* [09:33] Jeroen__u: ok. you can probably back even more off; the request:grant ratio is currently 7.5:1 [09:34] yipdw: It should go down over the next few minutes, the VMs have been shut down recently. [09:35] I was going to say if you need to have a gazillion instances running for some reason you could throw some of it at URLTeam [09:35] yipdw: I might do that, I hate URL shorteners. [09:36] So, who is the vantec guy? He is hogging a lot of jobs. [09:36] not here right now [09:40] job hogging isn't really a big problem [09:40] vantec might be running a lot of jobs but their system has also not checked in ~9100 items [09:40] it'll just get requeued and in time will be routed to a more modest, more reliable setup [09:41] Could you explain it a bit further? [09:41] I don't quite understand it. [09:41] each work item covers a certain article range [09:42] it can either succeed or fail; failed ones are detected by "hmm this hasn't reported back in a few days but we expect it to come back within an hour or two" [09:43] those that fail are thrown back into the queue and eventually they'll succeed, either by getting onto a system that won't eat itself or via script updates to e.g. handle some anomalous situation that shows up in that work item [09:43] Do not worry, I have shut down my workers gracefully. [09:44] Jeroen__u: you can just run the URLTeam scripts [09:44] There are a couple of other projects coming up too [09:45] yipdw: how can I take many articles without making enemies here? [09:45] You won't make enemies :) [09:45] you can't and that's kind of the point [09:45] We need the resources at the moment... The tracker prevents one person hogging the whole lot [09:45] Igloo^: I will run the URLTeam scripts too, but nujij is priority for me as I used to look at conversations over there. [09:46] =] [09:47] yipdw: What is the current request:grant ratio? I have shut down many jobs. [09:47] I have 48 left. [09:47] 3:1 [09:47] that's global [09:53] Global meaning over all projects? [09:53] sorry, global meaning across all requests on nujij [09:54] *** DrKyonko has quit IRC (Quit: Depression is merely anger without enthusiasm) [09:57] *** BlueMaxim has quit IRC (Quit: Leaving) [10:19] *** AlexLehm has joined #archiveteam [11:09] Jeroen__u: you can run yahooanswers. [11:09] We need a lot of IPs on that project, the more IPs the better [11:09] 2 concurrent per IP is adviced [11:09] I read about that. [11:18] *** irl has joined #archiveteam [11:18] *** zenguy_pc has quit IRC (Ping timeout: 260 seconds) [11:53] *** dashcloud has joined #archiveteam [12:24] *** tuankiet has quit IRC (Ping timeout: 244 seconds) [12:24] *** tuankiet has joined #archiveteam [12:53] *** ats has quit IRC (Read error: Operation timed out) [13:06] *** Whopper has quit IRC (Ping timeout: 370 seconds) [13:12] *** ats has joined #archiveteam [13:12] *** brayden has quit IRC (Read error: Operation timed out) [13:13] *** ky0ko has quit IRC (Read error: Operation timed out) [13:21] *** ky0ko has joined #archiveteam [13:26] *** brayden has joined #archiveteam [13:26] *** swebb sets mode: +o brayden [13:36] *** dashcloud has quit IRC (Read error: Operation timed out) [13:41] arkiver: on it. i hope the bug has been fixed [13:41] err, no [13:41] sorry, forgot to take a look [13:42] ah np [13:42] will do now [13:42] you should still have it in your query [13:42] thanks [13:42] yeah, got it in my logs [13:46] fixed. [13:53] *** ravetcofx has quit IRC (Ping timeout: 370 seconds) [14:00] I know that I'm new here, but I suggest that the nujij maximum requests per minute to be raised from 70 to 100 during 1 AM Europe/Amsterdam time to 6 AM Europe/Amsterdam time. [14:43] *** WinterFox has quit IRC (Read error: Operation timed out) [14:47] *** atomotic has joined #archiveteam [15:34] *** ravetcofx has joined #archiveteam [15:47] *** ravetcofx has quit IRC (Read error: Operation timed out) [15:49] *** atomotic has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…) [15:52] *** ravetcofx has joined #archiveteam [16:02] *** JesseW has joined #archiveteam [16:11] *** metal_cam has joined #archiveteam [16:14] *** BartoCH has quit IRC (Ping timeout: 260 seconds) [16:14] *** kristian_ has joined #archiveteam [16:31] *** JesseW has quit IRC (Ping timeout: 370 seconds) [16:39] *** atomotic has joined #archiveteam [16:47] *** JW_work1 has joined #archiveteam [16:48] *** BartoCH has joined #archiveteam [16:49] *** JW_work has quit IRC (Read error: Operation timed out) [16:51] *** JW_work1 has quit IRC (Client Quit) [17:13] *** RichardG has joined #archiveteam [17:13] *** RichardG has quit IRC (Client Quit) [17:13] *** RichardG has joined #archiveteam [17:34] *** Simpbrain has quit IRC (Ping timeout: 370 seconds) [17:34] *** Simpbrain has joined #archiveteam [17:37] *** bRick5772 has joined #archiveteam [17:53] *** Morbus has quit IRC (Read error: Operation timed out) [17:58] *** jeroen52_ has joined #archiveteam [17:59] *** Jeroen__u has quit IRC (Ping timeout: 268 seconds) [18:01] *** ravetcofx has quit IRC (Read error: Operation timed out) [18:04] *** RichardG_ has joined #archiveteam [18:04] *** RichardG has quit IRC (Read error: Operation timed out) [18:09] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [18:19] *** RichardG_ has quit IRC (Ping timeout: 370 seconds) [18:23] *** tuankiet has quit IRC (Ping timeout: 244 seconds) [18:36] *** tuankiet has joined #archiveteam [18:54] *** JW_work has joined #archiveteam [19:19] *** metalcamp has joined #archiveteam [19:22] *** metal_cam has quit IRC (Read error: Operation timed out) [19:29] *** JW_work1 has joined #archiveteam [19:31] *** JW_work has quit IRC (Read error: Operation timed out) [19:37] *** JW_work1 has quit IRC (Read error: Operation timed out) [19:42] *** ndiddy has joined #archiveteam [19:43] *** maseck has quit IRC (Remote host closed the connection) [19:51] *** maseck has joined #archiveteam [20:16] *** metalcamp has quit IRC (Read error: Operation timed out) [20:16] *** BartoCH has quit IRC (Ping timeout: 260 seconds) [20:18] *** BartoCH has joined #archiveteam [20:32] *** Jeroen52 has quit IRC (Read error: Connection reset by peer) [20:39] has anyone successfully tried to run warrior-dockerfile on a raspberry pi? just tried with latest hypriot image but container fails to start... [20:43] *** dashcloud has joined #archiveteam [20:44] bRick5772: i tried, docker is only x86_64 [20:45] yes, it used to be - but docker can be build on arm (since docker 1.10) [20:46] and that's what the hypriot guys did - build docker for arm based on raspbian and release a docker-prepared image: http://blog.hypriot.com/getting-started-with-docker-on-your-arm-device/ [20:48] mhh, so you'd need your own dockerfile then, is it? I guess the current one has some specific x86_64 binaries in there. [20:48] but i guess the stuff that's being loaded in the warrior-dockerfile is not arm-compatible - just wondering if someone succeeded with a raspi running the warrior-dockerfile [20:48] ok [20:48] you'd better run yourself the grab script in a daemonized screen session [20:49] those are written in python afaik, it should work [20:49] the logfile says something about"fatal error: rt_sigaction failure" when calling some .go files? [20:51] hmm, ok - well I like the idea of having a raspi sitting around just being dedicated to archiving-tasks and being supplied with jobs automatically [21:10] *** Jeroen52 has joined #archiveteam [21:11] *** ky0ko has quit IRC (Ping timeout: 244 seconds) [21:12] *** arkiver2 has joined #archiveteam [21:21] bRick5772: if you're up for the work, there's nothing stopping you from actually using the warrior scripts, I don't hink [21:21] it's pretty much python all the way down, iirc [21:22] http://archiveteam.org/index.php?title=Dev/Warrior#Bootup [21:32] *** schbirid has quit IRC (Quit: Leaving) [21:33] Kaz, thx I will give it a go - just tried to simply run boot.sh on the pi but there were some errors as well - I'll try to get it running and will let you know if there's something to improve [21:34] after all I think raspis are great hardware for such tasks... [21:36] raspis are not really that great for heavy IO stuffs, at least the one i have which is kinda becoming old [21:36] USB and ethernet are using a shared bus, this can really become problematic [21:38] but i'd be interested to hear if I'm wrong, my pi is really old now and they may have mitigated the problem [21:39] -bs alarm [21:39] *** ky0ko has joined #archiveteam [21:43] *** RichardG has joined #archiveteam [21:49] BartoCH: afaik the newer raspis (i got a raspi2) still share ethernet and usb on the same bus - but considering me being online via cable (30mbit) I guess the raspi will be able to handle the data. anyway, I'll let you know how it goes. [21:59] *** BlueMaxim has joined #archiveteam [22:11] *** bRick5772 has quit IRC (Quit: Leaving.) [22:14] *** RichardG has quit IRC (Read error: Operation timed out) [22:18] *** bsmith093 has joined #archiveteam [22:24] *** Honno has quit IRC (Read error: Operation timed out) [22:52] *** maelstrom has joined #archiveteam [23:05] *** WinterFox has joined #archiveteam [23:10] *** jdude104 has quit IRC (Ping timeout: 268 seconds) [23:55] *** melas has joined #archiveteam