[00:01] *** dashcloud has quit IRC (Read error: Operation timed out) [00:05] *** dashcloud has joined #archiveteam [00:14] *** bentpins has joined #archiveteam [00:25] *** philpem has quit IRC (Ping timeout: 252 seconds) [00:29] *** Start has joined #archiveteam [00:49] *** wvdp___ has quit IRC (Read error: Operation timed out) [01:06] *** bentpins has quit IRC (Ping timeout: 483 seconds) [01:08] *** godane has quit IRC (Quit: Leaving.) [01:08] *** godane has joined #archiveteam [01:09] *** JesseW has joined #archiveteam [01:14] *** primus104 has quit IRC (Leaving.) [01:43] *** signius has quit IRC (Read error: Operation timed out) [01:49] *** dashcloud has quit IRC (Read error: Operation timed out) [01:50] *** pfallenop has quit IRC (Ping timeout: 606 seconds) [01:53] *** dashcloud has joined #archiveteam [01:56] *** signius has joined #archiveteam [02:26] *** vitzli has joined #archiveteam [02:44] *** logan has quit IRC (Read error: Operation timed out) [02:50] *** bentpins has joined #archiveteam [03:32] *** xk_id has joined #archiveteam [03:34] *** logan has joined #archiveteam [03:39] *** xk_id has quit IRC (Remote host closed the connection) [03:40] *** xk_id has joined #archiveteam [03:57] *** BlueMaxim has joined #archiveteam [03:59] *** vitzli has quit IRC (Quit: Leaving) [04:05] *** dashcloud has quit IRC (Read error: Operation timed out) [04:10] *** dashcloud has joined #archiveteam [04:11] *** aaaaaaaaa has quit IRC (Leaving) [04:16] *** xk_id has quit IRC (Remote host closed the connection) [04:24] *** xk_id has joined #archiveteam [04:30] *** xk_id has quit IRC (Ping timeout: 369 seconds) [04:31] *** JesseW has quit IRC (Read error: Operation timed out) [04:36] *** JesseW has joined #archiveteam [04:49] https://code.stypi.com/e9fbieg3 [04:49] code pasting site [04:49] shutting down [04:49] september 3 [04:49] 8 chars lowercase alphanumeric IDs [04:53] *** khaoohs has quit IRC (Read error: Operation timed out) [04:53] *** JesseW has quit IRC (Leaving.) [04:56] *** xk_id has joined #archiveteam [04:58] *** Emcy_ has quit IRC (Ping timeout: 483 seconds) [05:00] They don't seem sequential, maybe it'd be easiest just asking nicely for a dump? [05:00] *** khaoohs has joined #archiveteam [05:05] *** xk_id has quit IRC (Read error: Operation timed out) [05:15] *** xk_id has joined #archiveteam [05:21] *** Start_ has joined #archiveteam [05:21] *** Start has quit IRC (Write error: Broken pipe) [05:26] *** espes__ has quit IRC (Quit: leaving) [05:28] *** espes__ has joined #archiveteam [05:28] *** espes__ has quit IRC (Client Quit) [05:28] *** espes__ has joined #archiveteam [05:33] *** dashcloud has quit IRC (Quit: No Ping reply in 180 seconds.) [05:34] *** dashcloud has joined #archiveteam [05:44] *** Start_ is now known as Start [05:49] *** MorbusIff has quit IRC (Read error: Operation timed out) [06:02] *** JesseW has joined #archiveteam [06:07] *** Start has quit IRC (Quit: Disconnected.) [06:08] *** Start has joined #archiveteam [06:22] *** JesseW has quit IRC (Ping timeout: 600 seconds) [06:28] *** pfallenop has joined #archiveteam [06:43] *** JesseW has joined #archiveteam [06:51] *** primus104 has joined #archiveteam [06:59] joepie91: added a wiki page: http://archiveteam.org/index.php?title=Stypi [07:07] Is this BLANK for everyone else also ? https://web.archive.org/web/20120714051606/http://ftp.sun.com/ [07:09] *** JesseW has quit IRC (Read error: Operation timed out) [07:13] *** PurpleSym has joined #archiveteam [07:51] signius: redirected me to here: https://web.archive.org/web/20120714051606/http://supportfiles.sun.com/warning [07:53] http://cdn.meme.am/instances/500x/57931550.jpg [07:53] errrr copypaste file. [07:53] Told me it was 302 and redirected. [08:00] *** asdf has joined #archiveteam [08:02] *** caber has joined #archiveteam [08:13] *** vitzli has joined #archiveteam [08:35] *** primus104 has quit IRC (Leaving.) [08:42] *** brayden has joined #archiveteam [08:42] *** swebb sets mode: +o brayden [08:42] *** vitzli has quit IRC (Quit: Leaving) [09:03] *** schbirid has joined #archiveteam [09:33] *** Stiletto has quit IRC (Read error: Operation timed out) [10:17] *** xk_id has quit IRC (Remote host closed the connection) [10:18] *** xk_id has joined #archiveteam [10:23] i am grabbing ftp.gameaholic.com [10:28] currently we're doing around 2000 items per minute in blingee. We need to get that to 10000 items per minute if we want to finish in time. [10:29] Deadline of blingee is august 25th. [10:29] #tragedee [10:30] *** xk_id has quit IRC (Read error: Operation timed out) [10:30] rsync is dead... [10:31] Currently for blip, yes, not for blingee [10:31] ah [10:31] k [10:31] shall I change to blingee? [10:31] for now that would be a good idea [10:42] *** primus104 has joined #archiveteam [10:51] *** Morbus has joined #archiveteam [11:39] *** deathy has quit IRC (Ping timeout: 252 seconds) [12:09] *** Wyatts has quit IRC (Remote host closed the connection) [12:12] *** Wyatts has joined #archiveteam [12:15] ftp://ftp:any@grandis.nu [12:18] *** mafrasi2 has quit IRC (No Ping reply in 180 seconds.) [12:19] *** mafrasi2 has joined #archiveteam [12:26] *** deathy has joined #archiveteam [12:48] *** deathy has quit IRC (Ping timeout: 252 seconds) [13:30] *** primus104 has quit IRC (Leaving.) [13:35] *** deathy has joined #archiveteam [13:39] *** Emcy_ has joined #archiveteam [13:44] *** brayden has quit IRC (Quit: Leaving) [13:50] *** PurpleSym has quit IRC (Remote host closed the connection) [13:56] *** wvdp___ has joined #archiveteam [14:03] *** mafrasi2 has quit IRC (No Ping reply in 180 seconds.) [14:03] *** zyphlar has quit IRC (Ping timeout: 252 seconds) [14:03] *** russss__ has quit IRC (Ping timeout: 252 seconds) [14:03] *** russss__ has joined #archiveteam [14:03] *** codl_ has quit IRC (Ping timeout: 252 seconds) [14:03] *** Ctrl-S has quit IRC (Ping timeout: 252 seconds) [14:03] *** kevin has quit IRC (Ping timeout: 252 seconds) [14:03] *** zyphlar has joined #archiveteam [14:04] *** codl_ has joined #archiveteam [14:04] arkiver: what's the bottleneck for blingee? [14:04] bandwidth / requests/ IPs? [14:04] *** Ctrl-S has joined #archiveteam [14:06] *** _0x2A has quit IRC (Ping timeout: 506 seconds) [14:06] *** kevin has joined #archiveteam [14:07] *** _0x2A has joined #archiveteam [14:08] *** altlabel has quit IRC (Ping timeout: 506 seconds) [14:14] *** BlueMaxim has quit IRC (Quit: Leaving) [14:17] *** altlabel has joined #archiveteam [14:17] *** dashcloud has quit IRC (Read error: Operation timed out) [14:20] *** dashcloud has joined #archiveteam [14:23] joepie91: currently bandwidth is (on our side) [14:24] or well, instances/concurrent on our side [14:24] arkiver: mmk. how crazy can I make it go [14:24] ie. how many concurrent [14:24] how crazy are you able to make it go? [14:25] I can always limit the number of items per minute on the tracker side so I think the more concurent the better [14:28] arkiver: I have a few bored servers, one of them is on 10gbps [14:28] lol [14:28] would you be able to add a few hundred concurrent? [14:28] maybe start with like 200 [14:32] *** dashcloud has quit IRC (Read error: Operation timed out) [14:34] *** mafrasi2 has joined #archiveteam [14:36] *** dashcloud has joined #archiveteam [14:37] arkiver: can certainly try - what's the avg disk usage I should expect per concurrent thread? [14:37] I have no idea, I'm not running the project at the moment [14:37] some people in #tragedee might know [14:42] joepie91: small, <20MB [14:55] db48x: noted, thanks [14:55] *** PurpleSym has joined #archiveteam [15:32] *** JesseW has joined #archiveteam [15:36] *** deathy has quit IRC (Remote host closed the connection) [15:40] *** deathy has joined #archiveteam [16:08] *** JesseW has quit IRC (Leaving.) [16:10] *** desuu has joined #archiveteam [16:10] Hey so im interested in installing my own archivebot instance for my own use [16:11] So I cloned the repo and setup the backend according to the install instructions, but where can I find the dashboard? I've tried http://localhost:4567/ but it doesnt seem like anything is listening on that port [16:11] *** Nertsy has joined #archiveteam [16:12] also I have no idea what the pipeline installation is for. Can anyone help me? [16:18] paging yipdw chfoo [16:18] desuu: might take a while before you get a response [16:18] :P [16:19] no problem [16:20] my connection is horrible though so just leave a message even if im not here [16:20] ive got a log bot so ill see it :) [16:21] *** desuu has quit IRC (Quit: Leaving) [16:30] you might try https://github.com/ludios/grab-site which is more geared towards local use [16:33] desuu: iirc pipeline is for the archivebot workers [16:33] archivebot backend pushes jobs to the workers [16:36] *** RichardG has joined #archiveteam [16:47] *** Morbus has quit IRC (Ping timeout: 252 seconds) [16:52] *** Morbus has joined #archiveteam [17:13] *** primus104 has joined #archiveteam [17:15] https://thedrumclub.bandcamp.com/album/kit-004-the-mega-bundle [17:15] *** philpem has joined #archiveteam [17:28] *** chfoo has quit IRC (Ping timeout: 258 seconds) [17:30] *** Stiletto has joined #archiveteam [17:42] *** chfoo has joined #archiveteam [17:45] *** primus104 has quit IRC (Leaving.) [17:53] *** aaaaaaaaa has joined #archiveteam [17:53] *** swebb sets mode: +o aaaaaaaaa [18:12] *** Morbus has quit IRC (Quit: http://www.disobey.com/) [18:15] *** Morbus has joined #archiveteam [18:19] *** khaoohs has quit IRC (Read error: Connection reset by peer) [18:20] *** khaoohs has joined #archiveteam [18:23] *** wvdp_ has joined #archiveteam [18:23] *** khaoohs_ has joined #archiveteam [18:25] *** khaoohs has quit IRC (Read error: Operation timed out) [18:27] *** khaoohs__ has joined #archiveteam [18:28] *** khaoohs_ has quit IRC (Read error: Operation timed out) [18:28] *** wvdp___ has quit IRC (Read error: Operation timed out) [18:51] *** primus104 has joined #archiveteam [18:52] *** dashcloud has quit IRC (Quit: No Ping reply in 180 seconds.) [18:54] *** dashcloud has joined #archiveteam [18:56] *** dashcloud has quit IRC (Read error: Operation timed out) [18:58] *** dashcloud has joined #archiveteam [19:19] *** RichardG has quit IRC (Remote host closed the connection) [20:09] *** wvdp_ has quit IRC (Read error: Operation timed out) [20:11] *** beardicus has quit IRC (Ping timeout: 483 seconds) [20:32] *** RichardG has joined #archiveteam [20:45] I'm sorry, User. I'm afraid I can't do that. [20:45] Please limit --concurrent to 20 or lower to avoid exhausting resources or triggering bugs. [20:45] @ db48x arkiver [20:45] tried to run it at lots [20:45] :p [20:45] *** habi has joined #archiveteam [20:46] *** habi has left [20:48] pipeline is the gnome3 of download scripts [20:55] *** joepie91 has quit IRC (Ping timeout: 252 seconds) [20:59] *** armitage_ has joined #archiveteam [20:59] *** armitage_ has quit IRC (Client Quit) [21:00] *** PurpleSym has quit IRC (Remote host closed the connection) [21:01] *** asdf has quit IRC (http://www.mibbit.com ajax IRC Client) [21:02] you can clone into another directory and start another pipeline [21:07] *** joepie91 has joined #archiveteam [21:13] *** beardicus has joined #archiveteam [21:33] *** asdf has joined #archiveteam [23:01] sounds like the ashleymadison data dump is at http://yuc3i3hat65rpl7t.onion/stuff/DarkHactivism.txt or magnet:?xt=urn:btih:40ae8a90de40ca3afa763c8edb43fc1fc47d75f1&dn=dmps fwiw [23:24] joepie91: yep [23:24] actually in this case I was happy for the reminder [23:24] db48x: can I convince it otherwise? [23:24] :p [23:24] forgot I was running it on a machine with a lower amount of ram than normal :P [23:25] sure, edit it [23:25] you could but seesaw gets weird at high concurrencies [23:26] pip install -e is your friend here [23:26] easier to just run multiple copies [23:27] db48x: are these all the items...? [23:27] these? [23:28] db48x: in the tracker currently [23:28] ah, no [23:28] I don't know how many arkiver said that there were [23:29] hrm, ascom is catching up [23:29] he is usually good at having an items repo [23:30] yep: https://github.com/ArchiveTeam/blingee-items/tree/master/ADDED [23:30] of course, that one doesn't have a todo folder [23:31] for just the blingees, it's >1,300,000 items [23:31] how much total size, est? [23:32] at least 25tb iirc [23:32] well, the average is 14MB each so far [23:32] there's also 22.5M users, thousands of groups, etc. [23:32] http://archive.fart.website/bin/irclogger_log/archiveteam?date=2015-08-16,Sun&sel=72#l68 [23:32] he estimated 130 TB there [23:33] actually 150 TB [23:33] mmk [23:33] we're only getting one size for each blingee. [23:33] it's huge, no matter what. [23:34] wait, 14MB * 130 million blingees is on the order of 2 PB [23:35] each item is 100 blingees. [23:37] balrog: is this an actual dump, or just one that someone is claiming came from Ashley Madison? [23:38] ok, so around 20 TB, that is a bit more sane [23:39] ok, I thought IA was going to have to pivot into the animated gif business [23:39] heh [23:39] lol [23:40] archivee.org [23:41] *** BlueMaxim has joined #archiveteam [23:54] http://www.wired.com/2015/08/happened-hackers-posted-stolen-ashley-madison-data/ [23:59] *** asdf has quit IRC (http://www.mibbit.com ajax IRC Client)