[00:02] arkiver: Could you requeue Yahoo Answers? [00:09] *** kristian_ has quit IRC (Remote host closed the connection) [00:10] done! [00:10] Thanks [00:14] *** cadbury_ has joined #archiveteam [00:16] *** Petri152 has quit IRC (Read error: Operation timed out) [00:19] *** nwf_ has quit IRC (Read error: Operation timed out) [00:20] *** Petri152 has joined #archiveteam [00:22] *** JesseW has joined #archiveteam [00:23] *** BlueMaxim has joined #archiveteam [00:30] *** phuzion has quit IRC (Ping timeout: 611 seconds) [00:32] *** Petri152 has quit IRC (Ping timeout: 613 seconds) [00:33] *** maelstrom has joined #archiveteam [00:42] *** Petri152 has joined #archiveteam [00:44] *** nwf_ has joined #archiveteam [00:45] *** phuzion has joined #archiveteam [01:00] *** octarine has quit IRC (Ping timeout: 260 seconds) [01:03] *** hook54321 has quit IRC (Ping timeout: 260 seconds) [01:04] *** _desu___ has quit IRC (Ping timeout: 260 seconds) [01:12] *** SmileyG has quit IRC (Ping timeout: 255 seconds) [01:15] *** Smiley has joined #archiveteam [01:47] *** balrog has quit IRC (Read error: Operation timed out) [01:47] *** Atros has joined #archiveteam [01:48] *** ranma has quit IRC (Read error: Operation timed out) [01:48] *** Mayonaise has quit IRC (Read error: Operation timed out) [01:48] *** FluffyFox has joined #archiveteam [01:48] *** aMunster has joined #archiveteam [01:48] *** SadDM has quit IRC (Read error: Operation timed out) [01:48] *** Frogging has quit IRC (Read error: Operation timed out) [01:48] *** atrocity has quit IRC (Read error: Operation timed out) [01:48] *** FluffyFox is now known as Frogging [01:49] *** marvinw has quit IRC (Ping timeout: 246 seconds) [01:49] *** remsen has quit IRC (Read error: Operation timed out) [01:49] *** computerf has quit IRC (Read error: Operation timed out) [01:49] *** acridAxid has quit IRC (Ping timeout: 246 seconds) [01:49] *** oli has quit IRC (Ping timeout: 246 seconds) [01:49] *** MMovie has joined #archiveteam [01:49] *** oli has joined #archiveteam [01:50] *** balrog has joined #archiveteam [01:50] *** swebb sets mode: +o balrog [01:50] *** kremlin has joined #archiveteam [01:50] *** TC01 has quit IRC (Ping timeout: 246 seconds) [01:50] *** yakfish has quit IRC (Ping timeout: 246 seconds) [01:50] *** robink has quit IRC (Ping timeout: 246 seconds) [01:50] *** robink has joined #archiveteam [01:52] *** jspiros has quit IRC (Read error: Operation timed out) [01:53] *** Atom has quit IRC (Read error: Operation timed out) [01:53] *** marvinw has joined #archiveteam [01:54] *** remsen has joined #archiveteam [01:54] *** kyounko has quit IRC (Ping timeout: 492 seconds) [01:55] *** TC01 has joined #archiveteam [01:56] *** cf has joined #archiveteam [01:58] *** ranma has joined #archiveteam [02:00] *** acridAxid has joined #archiveteam [02:01] *** Swizzle has quit IRC (Quit: Leaving) [02:05] *** Mayonaise has joined #archiveteam [02:06] *** JesseW has quit IRC (Ping timeout: 370 seconds) [02:33] *** Atros has quit IRC (Read error: Operation timed out) [02:33] *** atrocity has joined #archiveteam [02:56] *** JesseW has joined #archiveteam [03:31] *** xhdr has quit IRC (Ping timeout: 194 seconds) [03:31] *** Deewiant has quit IRC (Ping timeout: 194 seconds) [03:31] *** Sk1d has quit IRC (Ping timeout: 194 seconds) [03:31] *** SilSte has quit IRC (Ping timeout: 194 seconds) [03:31] *** xhdr has joined #archiveteam [03:31] *** Deewiant has joined #archiveteam [03:31] *** Sk1d has joined #archiveteam [03:37] *** SilSte has joined #archiveteam [04:03] *** ndiddy has quit IRC (Ping timeout: 244 seconds) [04:26] *** maelstrom has quit IRC (Quit: Leaving) [04:36] *** Sk1d has quit IRC (Ping timeout: 250 seconds) [04:37] *** VonGuard_ is now known as VonGuard [04:43] *** Sk1d has joined #archiveteam [05:28] *** hive-mind has quit IRC (Ping timeout: 260 seconds) [05:29] *** hive-mind has joined #archiveteam [05:59] *** JesseW has quit IRC (Ping timeout: 370 seconds) [07:03] *** AlexLehm has joined #archiveteam [07:06] SketchCow: grabbed [07:09] *** AlexLehm has quit IRC (Ping timeout: 260 seconds) [07:47] so I've increased my speed archiving genweb by using the wayback system to retrieve unique pages, cache them, then snarf them from wayback .. if I do it direct after a while I get tossed to /dev/null like I'm banned .. seems like I've got a bit to go .. Oct 1: 40172775 urls enumerated, 5861160 downloaded and scraped for more urls. Oct 5: 41830195 urls enumerated, 6142773 downloaded and scraped for more [07:47] urls. from 85.410% ... [07:47] ... unscraped to 85.315% unscraped. at some point enumerated urls will stop changing; I have one process priming the wayback cache, and another downloading, storing, scraping urls .. moved from sqlite3 to postgresql .. not sure how to make this go faster other than perhaps another pair of processes (that I'd have to do some form of overlap avoidance with) and I fear I may overload the genweb site if I [07:47] do too much more. I'm ... [07:47] ... hoping it stays online until I'm done ;-) [07:48] *** WinterFox has joined #archiveteam [07:50] *** Igloo^_^ has quit IRC (Ping timeout: 260 seconds) [07:54] *** atomotic has joined #archiveteam [07:57] *** schbirid has joined #archiveteam [08:14] *** dashcloud has quit IRC (Read error: Operation timed out) [08:21] *** MMovie1 has joined #archiveteam [08:22] *** MMovie has quit IRC (Read error: Operation timed out) [08:34] *** dashcloud has joined #archiveteam [09:00] *** ndiddy has joined #archiveteam [09:02] *** ravetcofx has quit IRC (Read error: Operation timed out) [09:03] *** Igloo^_^ has joined #archiveteam [10:25] *** godane has quit IRC (Read error: Operation timed out) [10:34] *** godane has joined #archiveteam [10:44] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [11:09] *** W1nterFox has joined #archiveteam [11:12] *** WinterFox has quit IRC (Ping timeout: 370 seconds) [11:20] *** BlueMaxim has quit IRC (Quit: Leaving) [11:22] *** Morbus has joined #archiveteam [11:58] *** atomotic has joined #archiveteam [13:41] *** samp has joined #archiveteam [13:45] Hello _o/ This is probably the wrong place to ask, but is there some ongoing project I can help with? I've been backuping my own pinboard links on my own using httrack and a friend of mine told me about archiveteam [14:01] *** JesseW has joined #archiveteam [14:17] *** Start has quit IRC (Quit: Disconnected.) [14:24] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [14:43] *** W1nterFox has quit IRC (Ping timeout: 370 seconds) [14:57] *** godane has quit IRC (Quit: Leaving.) [14:58] *** godane has joined #archiveteam [15:15] *** dashcloud has quit IRC (Quit: No Ping reply in 180 seconds.) [15:20] samp: the easierst way to help is running a warrior. Current main Project: yahoo answers. Please run with low concurrent(2), since yahoo is banning ips on more concurrent. [15:21] *** dashcloud has joined #archiveteam [15:45] *** JesseW has quit IRC (Quit: Leaving.) [15:45] *** JesseW has joined #archiveteam [15:51] Medowar0: been doing that but I chose "Archiveteam's choice", will change for Yahoo answers. I can help with code too but looks like there's no current project in need of one, right? [15:56] we really should change "Archiveteam's choice" to point at Yahoo Answers for now [15:59] *** atomotic has joined #archiveteam [16:00] this is the main comm channel, right? I'll keep it on radar to follow the group. For now I think I'll try to run the warrior on my RPi, it was kind of useless up until now [16:03] samp JesseW: AT Choice currently is URLTeam due to the IP Ban issue of yahoo. [16:04] samp: If you want to help with coding, start writing a detection system for bans into yahoo grab and make it pause then. Because right now, they retry every minute and fail every task and then proceed to request a new one. [16:05] also there are many ways to help. Check the wiki for that. [16:05] looks good, I'll look into it [16:07] *** JesseW has quit IRC (Ping timeout: 370 seconds) [16:10] *** ravetcofx has joined #archiveteam [16:11] *** lsmag has joined #archiveteam [16:12] *** lsmag has quit IRC (Client Quit) [16:23] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [16:41] *** Aoede has joined #archiveteam [17:04] *** hook54321 has joined #archiveteam [17:16] *** cadbury_ has quit IRC (Quit: leaving) [17:21] samp: wich languages do you know to code in? If you want you can help in #chaingang [17:24] luckcolor: Python+JS in my main job, I know some other languages but I can also learn sth new, if needed [17:25] luckcolor: what's chaingang? [17:25] wait [17:25] * luckcolor is getting the link from the wiki [17:26] sorry, I was doing the same thing [17:26] http://archiveteam.org/index.php?title=ArchiveTeam_Chain_Gang [17:27] to not flood this channel if you're still interested let's move the discussion to #chaingang [17:27] sure [17:33] *** Aoede has quit IRC (Quit: WeeChat 1.5) [17:37] *** cadbury_ has joined #archiveteam [18:43] *** AlexLehm has joined #archiveteam [19:01] *** BartoCH has joined #archiveteam [19:17] Do you have 4chan on your radar? [19:18] And i have a twitter user who I would like to save... its a private profile, but I can view it... [19:40] *** godane has quit IRC (Read error: Operation timed out) [19:41] *** godane has joined #archiveteam [19:58] *** bRick5772 has joined #archiveteam [20:10] *** maelstrom has joined #archiveteam [20:11] *** ete has joined #archiveteam [20:13] *** RichardG_ has joined #archiveteam [20:13] *** RichardG has quit IRC (Read error: Connection reset by peer) [20:13] sorry if this is the wrong place, but [20:13] http://news.berkeley.edu/2016/09/13/a-statement-on-online-course-content-and-accessibility/ "Federal government tells Berkeley they may not offer free online video courses, because they are discriminatory against deaf people who cannot hear the audio. Willing to reconsider if they translate them into sign language as well or add closed captioning, but the college says it can’t afford that and will probably just take the courses down. This is a metaphor for h [20:14] seems like something in danger / worth saving [20:17] *** ete has quit IRC (Remote host closed the connection) [20:33] *** bRick5772 has quit IRC (Ping timeout: 250 seconds) [20:34] *** bRick5772 has joined #archiveteam [21:15] *** BartoCH has quit IRC (Ping timeout: 260 seconds) [21:15] *** BartoCH has joined #archiveteam [21:18] *** schbirid has quit IRC (Quit: Leaving) [21:22] *** godane has quit IRC (Quit: Leaving.) [21:44] *** dashcloud has quit IRC (Read error: Operation timed out) [21:45] *** godane has joined #archiveteam [21:46] *** bRick5772 has quit IRC (Quit: Leaving.) [21:56] *** dashcloud has joined #archiveteam [22:06] *** dashcloud has quit IRC (Read error: Operation timed out) [22:09] *** dashcloud has joined #archiveteam [22:55] *** BartoCH has quit IRC (Quit: WeeChat 1.5) [22:58] *** AlexLehm has quit IRC (Ping timeout: 260 seconds) [23:01] *** tfgbd_znc has joined #archiveteam [23:16] *** RichardG_ has quit IRC (Ping timeout: 370 seconds) [23:20] ete (if you read the logs) — we know about it, but I'm not sure if there's an active effort to grab it. Help welcomed. [23:20] samp: URLTeam can also use help investigating shorteners; ask in #urlteam or check the wiki. [23:51] *** JesseW has joined #archiveteam