[00:07] *** BlueMaxim has quit IRC (Quit: Leaving) [00:08] *** BlueMaxim has joined #archiveteam-bs [00:30] *** Start has joined #archiveteam-bs [00:30] OK [00:31] oops, wrong window [01:00] thats what she said [01:00] I think wikipedia has bot detection. And for good reasons. [01:01] It limits the rate for bots, without disallowing them. [01:09] Blue... http://www.pratham.name/images/browser-by-country.png [01:15] this is cool- http://www.brendangregg.com/blog/2016-10-27/dtrace-for-linux-2016.html if you've got kernel 4.9rc1 or later, the same kind of raw tracing capabilities in Solaris are now in Linux [01:16] *** Swizzle has joined #archiveteam-bs [01:17] * Yoshimura suffers from second-system effect. [01:18] dashcloud: Kinda 'old' but excellent article. People that are doing multipath TCP and alike, or more specificaly devs benefit a lot from this. [01:19] I mean old thing. Excellent article. Two separate things. [01:56] anyone aware of a crc duplicate file checker where it saves results in a database to compare against (so I don't need to constantly have it scan all of my files each time it runs)? [01:56] I've tried Google and not finding one [01:58] *** tyzoid has joined #archiveteam-bs [01:58] tyzoid, works fine [01:58] jrwr: Glad to hear [01:58] I'm going to throw that box up on archive.org then [01:59] next would be cool to get a Windows Docker version of everything [01:59] Github should be able to take that file [01:59] as well [02:01] Swizzle: I can code you one. [02:01] Same here [02:01] clever burglar deterrent: https://youtu.be/3OBmprCm6mM?t=4m53s [02:02] I actually already made one, user code and error handling makes 2/3 - 3/4 non-library code sadly, and I did not finish that. [02:04] *** jrwr has quit IRC (Remote host closed the connection) [02:44] *** tyzoid has left [02:53] *** dashcloud has quit IRC (Read error: Connection reset by peer) [02:56] *** dashcloud has joined #archiveteam-bs [03:09] *** Swizzle has quit IRC (Read error: Operation timed out) [03:21] *** godane has quit IRC (Ping timeout: 255 seconds) [03:24] *** ndiddy has quit IRC (Read error: Connection reset by peer) [03:46] *** Yoshimura has quit IRC (Ping timeout: 255 seconds) [03:51] *** godane has joined #archiveteam-bs [04:09] *** ravetcofx has quit IRC (Read error: Operation timed out) [04:09] *** Whopper has quit IRC (Read error: Operation timed out) [04:11] *** Whopper has joined #archiveteam-bs [04:20] *** ravetcofx has joined #archiveteam-bs [04:23] *** godane has quit IRC (Quit: Leaving.) [04:24] *** godane has joined #archiveteam-bs [04:25] *** dashcloud has quit IRC (Read error: Operation timed out) [04:28] *** dashcloud has joined #archiveteam-bs [04:37] *** dashcloud has quit IRC (Read error: Operation timed out) [04:40] *** dashcloud has joined #archiveteam-bs [04:56] *** Madthias- has quit IRC (Quit: ▒^٥ ▒^٥) [04:58] *** Yoshimura has joined #archiveteam-bs [04:58] *** Swizzle has joined #archiveteam-bs [05:13] *** Start has quit IRC (Quit: Disconnected.) [05:15] *** Start has joined #archiveteam-bs [05:18] *** Sk1d has quit IRC (Ping timeout: 250 seconds) [05:24] *** Medowar0 has quit IRC (Remote host closed the connection) [05:26] *** Sk1d has joined #archiveteam-bs [05:33] *** Medowar0 has joined #archiveteam-bs [05:42] *** Aranje has quit IRC (Quit: Three sheets to the wind) [06:06] *** Swizzle has quit IRC (Quit: Leaving) [06:48] *** BartoCH has quit IRC (Quit: WeeChat 1.6) [06:51] *** BartoCH has joined #archiveteam-bs [07:27] *** GE has joined #archiveteam-bs [09:06] *** ravetcofx has quit IRC (Read error: Operation timed out) [09:32] *** brayden_ has joined #archiveteam-bs [09:32] *** swebb sets mode: +o brayden_ [09:37] *** brayden has quit IRC (Read error: Operation timed out) [12:20] *** BlueMaxim has quit IRC (Quit: Leaving) [12:38] *** VADemon has joined #archiveteam-bs [12:58] *** GE has quit IRC (Remote host closed the connection) [13:35] *** hewi has quit IRC (Ping timeout: 268 seconds) [14:10] *** VADemon has quit IRC (Quit: left4dead) [15:33] *** GE has joined #archiveteam-bs [15:55] *** zhongfu has quit IRC (Ping timeout: 260 seconds) [15:55] *** zhongfu has joined #archiveteam-bs [16:15] *** Start has quit IRC (Quit: Disconnected.) [16:26] *** Yoshimura has quit IRC (Remote host closed the connection) [16:27] *** Yoshimura has joined #archiveteam-bs [16:27] *** Yoshimura has quit IRC (Client Quit) [16:27] *** Yoshimura has joined #archiveteam-bs [18:12] http://www.nytco.com/the-new-york-times-to-offer-open-access-to-nytimes-com-november-7-9/ [18:12] "Readers will have unlimited access to NYTimes.com for 72 hours from 12:01 a.m. ET on Monday, November 7" [18:28] *** ravetcofx has joined #archiveteam-bs [18:35] *** REiN^ has joined #archiveteam-bs [18:47] so looks like kpfa downloads are going at normal speeds [18:47] there not rate limit it anymore [19:12] Great [19:21] https://twitter.com/PCBrown/status/794210799199760384 [19:31] *** JW_work has quit IRC (Quit: Leaving.) [19:32] *** JW_work has joined #archiveteam-bs [19:32] *** jrwr has joined #archiveteam-bs [19:56] *** JW_work has quit IRC (Read error: Operation timed out) [19:58] *** kristian_ has joined #archiveteam-bs [20:15] *** JW_work has joined #archiveteam-bs [20:41] *** RichardG_ has joined #archiveteam-bs [20:41] *** RichardG has quit IRC (Read error: Connection reset by peer) [20:55] *** RichardG has joined #archiveteam-bs [20:58] *** BlueMaxim has joined #archiveteam-bs [21:00] *** RichardG_ has quit IRC (Ping timeout: 370 seconds) [21:18] *** Stiletto has quit IRC (Ping timeout: 250 seconds) [22:04] *** robink has joined #archiveteam-bs [22:12] *** Stiletto has joined #archiveteam-bs [22:15] *** hewi_ has joined #archiveteam-bs [22:53] *** ravetcofx has quit IRC (Read error: Operation timed out) [22:54] *** Stiletto has quit IRC (Ping timeout: 260 seconds) [23:01] *** ravetcofx has joined #archiveteam-bs [23:06] *** GE has quit IRC (Remote host closed the connection) [23:11] *** acridAxid has quit IRC (Quit: marauder) [23:12] *** RichardG_ has joined #archiveteam-bs [23:12] *** acridAxid has joined #archiveteam-bs [23:16] Is there any pie chart / anylysis on how much data text / textlike resources vs binary data take? [23:17] DFJustin: how much of the collection is archivebot? [23:17] I would be really interested in that. Some textual stuff can be unbelievably shrunk. [23:17] *** RichardG has quit IRC (Read error: Operation timed out) [23:39] archivebot is 185,176,271,107 KB but I think that's not included in the first number because the name is archivebot and not archiveteam_archivebot etc.