[00:01] *** chfoo has quit IRC (chfoo) [00:07] *** chfoo has joined #archiveteam [00:25] *** kyan has quit IRC (Ping timeout: 258 seconds) [00:37] *** Start has joined #archiveteam [00:51] *** mistym has quit IRC (Remote host closed the connection) [00:53] *** signius has quit IRC (Read error: Operation timed out) [00:55] *** primus104 has quit IRC (Leaving.) [01:02] *** signius has joined #archiveteam [01:20] *** RedType has quit IRC (Quit: leaving) [01:20] *** RedType has joined #archiveteam [01:32] *** username1 has joined #archiveteam [01:34] *** schbirid2 has quit IRC (Read error: Operation timed out) [01:38] *** RedType has quit IRC (Quit: womp) [01:38] *** RedType has joined #archiveteam [01:42] *** mistym has joined #archiveteam [02:08] *** kyan has joined #archiveteam [02:38] *** Ymgve has quit IRC () [03:12] *** ikreymer has quit IRC (Remote host closed the connection) [03:15] *** dcmorton has quit IRC (Quit: ZNC - http://znc.in) [03:18] *** dcmorton has joined #archiveteam [03:38] *** VADemon has quit IRC (Quit: left4dead) [03:38] *** tsp_ has quit IRC (Ping timeout: 606 seconds) [03:39] *** tsp_ has joined #archiveteam [03:44] *** ikreymer has joined #archiveteam [04:19] *** antomatic has joined #archiveteam [04:20] *** aNthraXx has quit IRC (Read error: Operation timed out) [04:20] *** antomati_ has quit IRC (Read error: Operation timed out) [04:20] *** Sk1d has quit IRC (Read error: Operation timed out) [04:21] *** Sk1d has joined #archiveteam [04:21] *** cadbury_ has quit IRC (Read error: Operation timed out) [04:22] *** brayden has quit IRC (Read error: Operation timed out) [04:22] *** aaaaaaaaa has quit IRC (Leaving) [04:22] *** caber has quit IRC (Read error: Operation timed out) [04:24] *** ikreymer_ has joined #archiveteam [04:25] *** caber has joined #archiveteam [04:26] *** ikreymer has quit IRC (Read error: Operation timed out) [04:26] *** cadbury_ has joined #archiveteam [04:31] *** aNthraXx has joined #archiveteam [04:34] *** mistym has quit IRC (Remote host closed the connection) [04:35] *** aNthraXx has quit IRC (Read error: Operation timed out) [04:36] *** cadbury_ has quit IRC (Read error: Operation timed out) [04:41] *** cadbury_ has joined #archiveteam [04:42] *** brayden has joined #archiveteam [04:42] *** aNthraXx has joined #archiveteam [04:52] *** useretail has quit IRC (Dreaming in digital. Living in real-time. Thinking in binary. Talking in IP.) [05:02] *** mistym has joined #archiveteam [05:11] *** kyan has quit IRC (Quit: Leaving) [05:15] *** mrfoo has joined #archiveteam [05:18] *** ikreymer_ has quit IRC () [05:43] *** hive-mind has quit IRC (Ping timeout: 252 seconds) [05:43] *** wp494 has quit IRC (Ping timeout: 740 seconds) [06:04] *** wp494 has joined #archiveteam [06:08] *** hive-mind has joined #archiveteam [06:40] *** mistym has quit IRC (Remote host closed the connection) [07:37] *** primus104 has joined #archiveteam [07:41] *** mistym has joined #archiveteam [07:51] *** mistym has quit IRC (Ping timeout: 483 seconds) [07:53] *** wp494 has quit IRC (LOUD UNNECESSARY QUIT MESSAGES) [07:54] *** wp494 has joined #archiveteam [08:40] *** atomotic has joined #archiveteam [08:57] *** marvinw has quit IRC (Ping timeout: 252 seconds) [09:04] *** atomotic has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…) [09:09] *** marvinw has joined #archiveteam [09:49] *** primus104 has quit IRC (Leaving.) [10:33] *** Rotab has quit IRC (Ping timeout: 198 seconds) [10:35] *** Rotab has joined #archiveteam [11:20] *** mistym has joined #archiveteam [11:24] *** mistym has quit IRC (Ping timeout: 252 seconds) [11:44] *** Ymgve has joined #archiveteam [12:24] *** username1 is now known as schbirid [12:29] *** wp494 has quit IRC (Ping timeout: 740 seconds) [12:40] https://www.kickstarter.com/projects/2146199819/zx-spectrum-a-visual-compendium-by-bitmap-books all my money. all of it. [12:42] *** sankin has joined #archiveteam [12:47] *** BlueMaxim has quit IRC (Quit: Leaving) [13:12] *** primus104 has joined #archiveteam [13:13] Oh jeez [13:13] * ersi wipes mouth [13:41] ----------------------------- [13:41] REMINDER: FOS GOES DOWN FOR AN HOUR OR TWO [13:41] MAIN SYSTEM IS GETTING SOME UPGRADES [13:42] SHOULD RETURN IN NO TIME AND WILL RESUME [13:42] ----------------------------- [13:42] Time to down - about 3 hours [13:57] *** useretail has joined #archiveteam [14:02] *** philpem has joined #archiveteam [14:09] *** mistym has joined #archiveteam [14:22] *** mistym has quit IRC (Read error: Operation timed out) [14:31] *** mistym has joined #archiveteam [14:48] *** primus has joined #archiveteam [15:07] *** primus104 has quit IRC (Leaving.) [15:24] *** Stiletto has quit IRC () [15:35] *** Wilki has joined #archiveteam [15:35] hello [15:35] Who is uploading Atari books to the FTP dump? [15:38] !a http://rethinkvps.com/ [15:40] It looks like rethinkvps.com might be going under, whats the possibility of getting it archived for future reference? [15:40] *** mistym has quit IRC (Remote host closed the connection) [15:42] *** Wilki has quit IRC (Quit: Page closed) [15:49] What part of it? [15:54] *** [Beta] has quit IRC (Ping timeout: 240 seconds) [15:55] *** aaaaaaaaa has joined #archiveteam [16:12] *** Stiletto has joined #archiveteam [16:23] *** t4nk391 has joined #archiveteam [16:24] hey is anyone active here? [16:24] atm? [16:28] *** mistym has joined #archiveteam [16:40] things not to ask on irc #3 [16:41] can I ask a question? [16:42] I've got a question, is anyone here [16:42] yup [16:42] im here fire away [16:42] XD t4nk391 I'm just taking the piss cc schbirid [16:42] Jeez, just ask your question instead of asking to ask a question [16:42] ersi, i know right [16:54] *** SimpBrain has joined #archiveteam [17:03] for some reason that made me think of https://www.youtube.com/watch?v=rEmTMkx7SLE [17:03] *** philpem has quit IRC (Ping timeout: 252 seconds) [17:05] interesting fact, that was the first hit for "pink floyd is there anybody out there" rather than the song of the same name by them. [17:08] Playing Elastinen ft asa viidest ysiin-OUKbnDMHBXE.m4a. [17:08] whoops paste fail [17:09] https://www.youtube.com/watch?v=U13xOvDa19U [17:10] this better be good xD [17:11] oh yeah [17:12] that works amazingly well [17:13] strange right? [17:14] erm, yeah [17:14] wrong channel :-) [17:14] *** edsu_ is now known as edsu [17:14] there are some good mashups out there but there are also bad ones [17:14] => #-bs [17:18] Is anyone active here? [17:18] Archive Team: We're not archive.org | http://archiveteam.org/ | lengthy/off-topic in #archiveteam-bs | Is anyone active here? [17:19] ---- [17:19] whats the chance of a website kicking me if i crawl 644340 pages? trying to backup a site [17:19] no [17:19] *** SketchCow changes topic to: Archive Team: We're not archive.org | http://archiveteam.org/ | lengthy/off-topic in #archiveteam-bs | Is anyone active here? [17:19] ---- [17:19] t4nk391: depends on who runs it and what they decide [17:20] yeah, some crawls get into the millions of pages with no problems some only get a couple hundred before the hammer falls. [17:20] *** signius has quit IRC (Read error: Operation timed out) [17:21] I think the site was made in 2004, so i should be good until they check there logs [17:22] does it make any difference if they use cloudflare? would they kick me / ban IP? [17:23] ah, now we're approaching an answerable question [17:23] i don't think cloudflare does anti-crawling [17:23] do they pay for cloudflare? [17:24] i dont think so, they are having financial trouble. might shutdown on monday hense the crawl [17:24] go for it [17:25] cloudflare only does ddos protection on paid accounts, so if it did anti-crawling then it'd also only be for paid accounts [17:26] @xmc: legend - thanks for the replies [17:26] sure thing [17:27] do you need pointers on how to do a high-quality site crawl? [17:27] btw its a (crappy) private torrent tracker and i have 3 invites if anyone wants (bitsoup.me) [17:28] pointers? not really, i have so far logged in with my script [17:28] ok [17:29] if you're comfortable writing lua code to make the crawler happen, we recommend using wpull to do the bulk of the work [17:29] wpull lets you write site-specific lua scripts and does most of the rest of things itself [17:29] if you're interested: https://github.com/chfoo/wpull [17:30] oh nice, I wish i knew lua, using curl with php atm [17:30] lua's pretty straightforward [17:31] :) [17:31] here is, for example, a lua script to archive a phpbb3 forum https://github.com/ArchiveTeam/wget-lua-forum-scripts/blob/master/phpbb3.lua [17:31] I have thought about using lua on top of C/C++ to make a game [17:32] that is a common use for lua [17:32] Cloudflare does have crawler detection [17:33] or at least they offer it [17:33] *** signius has joined #archiveteam [17:33] ArchiveBot hits that occasionally so you will want to watch your WARCs [17:33] hm, ok, it sounds like a thing that they would charge for [17:34] oh ok, so use my vps, when crawling so i don't get a bad rep with cloudflare [17:34] there seems to be at least one Cloudflare employee that has starred the ArchiveBot repo. I'm not sure what that implies [17:34] huh. [17:35] t4nk391: we also have crawlers here, so unless you have a custom job and/or want to tinker around we can also run a crawl [17:37] *** primus104 has joined #archiveteam [17:37] @yipdw: ill keep that in mind [17:37] yipdw: sounds like site needs to be logged in to crawl [17:37] oh whee [17:37] so technically speaking it's out of scope for archiveteam proper [17:37] yes [17:37] being a private tracker [17:37] @xmc i've covered that in my script, it also needs https [17:37] * xmc nods [17:38] t4nk391: in irc you don't need to put an @ before someone's name to mention them. you can usually just type the first few characters of their name and press [17:39] xmc: im just copying and pasting also using the browser clieny [17:39] * xmc nods [17:39] no worries [17:44] Whats an easy way to store HTML for mining later? [17:44] WARC [17:45] we should take this to #archiveteam-bs [17:45] imo here is fine, but i'm not going to stop you [17:46] => #-bs is an archiveteam meme [17:46] should i stay or should i go? [18:08] yes [18:09] ersi is, as always, helpful [18:09] \:D/ [18:10] what a strange gesture [18:14] *** Nystrom has joined #archiveteam [18:16] Well im off to bed (6:19am here), have a good day/night all :D [18:20] *** t4nk391 has quit IRC (Quit: Page closed) [18:24] *** primus104 has quit IRC (Leaving.) [18:38] gitorious rsync: http://zeppelin.xrtc.net/corp.xrtc.net/karbovanets.corp.xrtc.net/if_eth0.html [18:38] eta 10 days [18:41] same graph, different machine https://vpssd.com/munin/vpssd.com/jimmy.vpssd.com/if_p2p1_10.html [18:56] *** Zebranky has quit IRC (Remote host closed the connection) [18:57] i'd like to stress that this is the first time I can remember since archiveteam started that someone has actually come to us proactively [18:57] since then i have graduated university, moved three times, and had four different jobs [18:58] -------------------------- [18:58] FOS.TEXTFILES.COM is back [18:58] -------------------------- [18:58] woooo [18:59] I'll get the crap going later today [19:00] *** Nertsy has quit IRC (Read error: Operation timed out) [19:01] SketchCow: halo is still on hold, iirc, do you want to restart it sometime [19:27] *** Nertsy has joined #archiveteam [19:27] *** primus104 has joined #archiveteam [19:34] *** Zebranky has joined #archiveteam [19:35] *** scyther has joined #archiveteam [19:50] *** lytv has quit IRC (Max SendQ exceeded) [19:50] *** Fusl has quit IRC (Max SendQ exceeded) [19:53] *** lytv has joined #archiveteam [19:55] *** SN4T14_ has joined #archiveteam [19:55] *** Fusl has joined #archiveteam [20:00] *** SN4T14__ has quit IRC (Ping timeout: 369 seconds) [20:03] *** SimpBrain has quit IRC (Quit: Leaving) [20:44] *** sankin has quit IRC (Leaving.) [20:44] *** mistym has quit IRC (Remote host closed the connection) [20:50] *** SimpBrain has joined #archiveteam [20:59] *** scyther has quit IRC (Leaving) [21:18] *** philpem has joined #archiveteam [21:25] *** Rotab has quit IRC (Read error: Connection reset by peer) [21:37] Yeah [21:37] I wonder what's filling it - lots of halo still coming in [21:40] *** SimpBrain has quit IRC (Quit: Leaving) [21:47] *** mistym has joined #archiveteam [21:49] *** Rotab has joined #archiveteam [21:51] *** pwnsrv has quit IRC (Ping timeout: 265 seconds) [21:52] hm, or is halo not on hold [21:52] I think we turned it on a little. [21:53] ok yeah it's letting thru 5 items per minute [22:12] *** pwnsrv has joined #archiveteam [22:38] *** mistym_ has joined #archiveteam [22:44] *** Emcy has quit IRC (Ping timeout: 512 seconds) [22:44] *** mistym has quit IRC (Read error: Operation timed out) [22:55] *** wp494 has joined #archiveteam [23:11] *** Mayonaise has quit IRC (Ping timeout: 362 seconds) [23:13] *** RedType has quit IRC (Quit: leaving) [23:14] *** RedType has joined #archiveteam [23:14] *** sirdancea has joined #archiveteam [23:16] *** RedType has quit IRC (Client Quit) [23:26] *** lysobit has quit IRC (Ping timeout: 512 seconds) [23:27] *** BlueMaxim has joined #archiveteam [23:27] *** RedType has joined #archiveteam [23:28] *** lysobit has joined #archiveteam [23:36] *** Mayonaise has joined #archiveteam [23:46] *** nertzy has joined #archiveteam