#archiveteam 2015-08-18,Tue

↑back Search

Time Nickname Message
00:01 🔗 dashcloud has quit IRC (Read error: Operation timed out)
00:05 🔗 dashcloud has joined #archiveteam
00:14 🔗 bentpins has joined #archiveteam
00:25 🔗 philpem has quit IRC (Ping timeout: 252 seconds)
00:29 🔗 Start has joined #archiveteam
00:49 🔗 wvdp___ has quit IRC (Read error: Operation timed out)
01:06 🔗 bentpins has quit IRC (Ping timeout: 483 seconds)
01:08 🔗 godane has quit IRC (Quit: Leaving.)
01:08 🔗 godane has joined #archiveteam
01:09 🔗 JesseW has joined #archiveteam
01:14 🔗 primus104 has quit IRC (Leaving.)
01:43 🔗 signius has quit IRC (Read error: Operation timed out)
01:49 🔗 dashcloud has quit IRC (Read error: Operation timed out)
01:50 🔗 pfallenop has quit IRC (Ping timeout: 606 seconds)
01:53 🔗 dashcloud has joined #archiveteam
01:56 🔗 signius has joined #archiveteam
02:26 🔗 vitzli has joined #archiveteam
02:44 🔗 logan has quit IRC (Read error: Operation timed out)
02:50 🔗 bentpins has joined #archiveteam
03:32 🔗 xk_id has joined #archiveteam
03:34 🔗 logan has joined #archiveteam
03:39 🔗 xk_id has quit IRC (Remote host closed the connection)
03:40 🔗 xk_id has joined #archiveteam
03:57 🔗 BlueMaxim has joined #archiveteam
03:59 🔗 vitzli has quit IRC (Quit: Leaving)
04:05 🔗 dashcloud has quit IRC (Read error: Operation timed out)
04:10 🔗 dashcloud has joined #archiveteam
04:11 🔗 aaaaaaaaa has quit IRC (Leaving)
04:16 🔗 xk_id has quit IRC (Remote host closed the connection)
04:24 🔗 xk_id has joined #archiveteam
04:30 🔗 xk_id has quit IRC (Ping timeout: 369 seconds)
04:31 🔗 JesseW has quit IRC (Read error: Operation timed out)
04:36 🔗 JesseW has joined #archiveteam
04:49 🔗 joepie91 https://code.stypi.com/e9fbieg3
04:49 🔗 joepie91 code pasting site
04:49 🔗 joepie91 shutting down
04:49 🔗 joepie91 september 3
04:49 🔗 joepie91 8 chars lowercase alphanumeric IDs
04:53 🔗 khaoohs has quit IRC (Read error: Operation timed out)
04:53 🔗 JesseW has quit IRC (Leaving.)
04:56 🔗 xk_id has joined #archiveteam
04:58 🔗 Emcy_ has quit IRC (Ping timeout: 483 seconds)
05:00 🔗 bentpins They don't seem sequential, maybe it'd be easiest just asking nicely for a dump?
05:00 🔗 khaoohs has joined #archiveteam
05:05 🔗 xk_id has quit IRC (Read error: Operation timed out)
05:15 🔗 xk_id has joined #archiveteam
05:21 🔗 Start_ has joined #archiveteam
05:21 🔗 Start has quit IRC (Write error: Broken pipe)
05:26 🔗 espes__ has quit IRC (Quit: leaving)
05:28 🔗 espes__ has joined #archiveteam
05:28 🔗 espes__ has quit IRC (Client Quit)
05:28 🔗 espes__ has joined #archiveteam
05:33 🔗 dashcloud has quit IRC (Quit: No Ping reply in 180 seconds.)
05:34 🔗 dashcloud has joined #archiveteam
05:44 🔗 Start_ is now known as Start
05:49 🔗 MorbusIff has quit IRC (Read error: Operation timed out)
06:02 🔗 JesseW has joined #archiveteam
06:07 🔗 Start has quit IRC (Quit: Disconnected.)
06:08 🔗 Start has joined #archiveteam
06:22 🔗 JesseW has quit IRC (Ping timeout: 600 seconds)
06:28 🔗 pfallenop has joined #archiveteam
06:43 🔗 JesseW has joined #archiveteam
06:51 🔗 primus104 has joined #archiveteam
06:59 🔗 JesseW joepie91: added a wiki page: http://archiveteam.org/index.php?title=Stypi
07:07 🔗 signius Is this BLANK for everyone else also ? https://web.archive.org/web/20120714051606/http://ftp.sun.com/
07:09 🔗 JesseW has quit IRC (Read error: Operation timed out)
07:13 🔗 PurpleSym has joined #archiveteam
07:51 🔗 dan- signius: redirected me to here: https://web.archive.org/web/20120714051606/http://supportfiles.sun.com/warning
07:53 🔗 Smiley http://cdn.meme.am/instances/500x/57931550.jpg
07:53 🔗 Smiley errrr copypaste file.
07:53 🔗 Smiley Told me it was 302 and redirected.
08:00 🔗 asdf has joined #archiveteam
08:02 🔗 caber has joined #archiveteam
08:13 🔗 vitzli has joined #archiveteam
08:35 🔗 primus104 has quit IRC (Leaving.)
08:42 🔗 brayden has joined #archiveteam
08:42 🔗 swebb sets mode: +o brayden
08:42 🔗 vitzli has quit IRC (Quit: Leaving)
09:03 🔗 schbirid has joined #archiveteam
09:33 🔗 Stiletto has quit IRC (Read error: Operation timed out)
10:17 🔗 xk_id has quit IRC (Remote host closed the connection)
10:18 🔗 xk_id has joined #archiveteam
10:23 🔗 schbirid i am grabbing ftp.gameaholic.com
10:28 🔗 arkiver currently we're doing around 2000 items per minute in blingee. We need to get that to 10000 items per minute if we want to finish in time.
10:29 🔗 arkiver Deadline of blingee is august 25th.
10:29 🔗 arkiver #tragedee
10:30 🔗 xk_id has quit IRC (Read error: Operation timed out)
10:30 🔗 Silvan rsync is dead...
10:31 🔗 arkiver Currently for blip, yes, not for blingee
10:31 🔗 Silvan ah
10:31 🔗 Silvan k
10:31 🔗 Silvan shall I change to blingee?
10:31 🔗 arkiver for now that would be a good idea
10:42 🔗 primus104 has joined #archiveteam
10:51 🔗 Morbus has joined #archiveteam
11:39 🔗 deathy has quit IRC (Ping timeout: 252 seconds)
12:09 🔗 Wyatts has quit IRC (Remote host closed the connection)
12:12 🔗 Wyatts has joined #archiveteam
12:15 🔗 schbirid ftp://ftp:any@grandis.nu
12:18 🔗 mafrasi2 has quit IRC (No Ping reply in 180 seconds.)
12:19 🔗 mafrasi2 has joined #archiveteam
12:26 🔗 deathy has joined #archiveteam
12:48 🔗 deathy has quit IRC (Ping timeout: 252 seconds)
13:30 🔗 primus104 has quit IRC (Leaving.)
13:35 🔗 deathy has joined #archiveteam
13:39 🔗 Emcy_ has joined #archiveteam
13:44 🔗 brayden has quit IRC (Quit: Leaving)
13:50 🔗 PurpleSym has quit IRC (Remote host closed the connection)
13:56 🔗 wvdp___ has joined #archiveteam
14:03 🔗 mafrasi2 has quit IRC (No Ping reply in 180 seconds.)
14:03 🔗 zyphlar has quit IRC (Ping timeout: 252 seconds)
14:03 🔗 russss__ has quit IRC (Ping timeout: 252 seconds)
14:03 🔗 russss__ has joined #archiveteam
14:03 🔗 codl_ has quit IRC (Ping timeout: 252 seconds)
14:03 🔗 Ctrl-S has quit IRC (Ping timeout: 252 seconds)
14:03 🔗 kevin has quit IRC (Ping timeout: 252 seconds)
14:03 🔗 zyphlar has joined #archiveteam
14:04 🔗 codl_ has joined #archiveteam
14:04 🔗 joepie91 arkiver: what's the bottleneck for blingee?
14:04 🔗 joepie91 bandwidth / requests/ IPs?
14:04 🔗 Ctrl-S has joined #archiveteam
14:06 🔗 _0x2A has quit IRC (Ping timeout: 506 seconds)
14:06 🔗 kevin has joined #archiveteam
14:07 🔗 _0x2A has joined #archiveteam
14:08 🔗 altlabel has quit IRC (Ping timeout: 506 seconds)
14:14 🔗 BlueMaxim has quit IRC (Quit: Leaving)
14:17 🔗 altlabel has joined #archiveteam
14:17 🔗 dashcloud has quit IRC (Read error: Operation timed out)
14:20 🔗 dashcloud has joined #archiveteam
14:23 🔗 arkiver joepie91: currently bandwidth is (on our side)
14:24 🔗 arkiver or well, instances/concurrent on our side
14:24 🔗 joepie91 arkiver: mmk. how crazy can I make it go
14:24 🔗 joepie91 ie. how many concurrent
14:24 🔗 arkiver how crazy are you able to make it go?
14:25 🔗 arkiver I can always limit the number of items per minute on the tracker side so I think the more concurent the better
14:28 🔗 joepie91 arkiver: I have a few bored servers, one of them is on 10gbps
14:28 🔗 joepie91 lol
14:28 🔗 arkiver would you be able to add a few hundred concurrent?
14:28 🔗 arkiver maybe start with like 200
14:32 🔗 dashcloud has quit IRC (Read error: Operation timed out)
14:34 🔗 mafrasi2 has joined #archiveteam
14:36 🔗 dashcloud has joined #archiveteam
14:37 🔗 joepie91 arkiver: can certainly try - what's the avg disk usage I should expect per concurrent thread?
14:37 🔗 arkiver I have no idea, I'm not running the project at the moment
14:37 🔗 arkiver some people in #tragedee might know
14:42 🔗 db48x joepie91: small, <20MB
14:55 🔗 joepie91 db48x: noted, thanks
14:55 🔗 PurpleSym has joined #archiveteam
15:32 🔗 JesseW has joined #archiveteam
15:36 🔗 deathy has quit IRC (Remote host closed the connection)
15:40 🔗 deathy has joined #archiveteam
16:08 🔗 JesseW has quit IRC (Leaving.)
16:10 🔗 desuu has joined #archiveteam
16:10 🔗 desuu Hey so im interested in installing my own archivebot instance for my own use
16:11 🔗 desuu So I cloned the repo and setup the backend according to the install instructions, but where can I find the dashboard? I've tried http://localhost:4567/ but it doesnt seem like anything is listening on that port
16:11 🔗 Nertsy has joined #archiveteam
16:12 🔗 desuu also I have no idea what the pipeline installation is for. Can anyone help me?
16:18 🔗 joepie91 paging yipdw chfoo
16:18 🔗 joepie91 desuu: might take a while before you get a response
16:18 🔗 joepie91 :P
16:19 🔗 desuu no problem
16:20 🔗 desuu my connection is horrible though so just leave a message even if im not here
16:20 🔗 desuu ive got a log bot so ill see it :)
16:21 🔗 desuu has quit IRC (Quit: Leaving)
16:30 🔗 DFJustin you might try https://github.com/ludios/grab-site which is more geared towards local use
16:33 🔗 zhongfu desuu: iirc pipeline is for the archivebot workers
16:33 🔗 zhongfu archivebot backend pushes jobs to the workers
16:36 🔗 RichardG has joined #archiveteam
16:47 🔗 Morbus has quit IRC (Ping timeout: 252 seconds)
16:52 🔗 Morbus has joined #archiveteam
17:13 🔗 primus104 has joined #archiveteam
17:15 🔗 schbirid https://thedrumclub.bandcamp.com/album/kit-004-the-mega-bundle
17:15 🔗 philpem has joined #archiveteam
17:28 🔗 chfoo has quit IRC (Ping timeout: 258 seconds)
17:30 🔗 Stiletto has joined #archiveteam
17:42 🔗 chfoo has joined #archiveteam
17:45 🔗 primus104 has quit IRC (Leaving.)
17:53 🔗 aaaaaaaaa has joined #archiveteam
17:53 🔗 swebb sets mode: +o aaaaaaaaa
18:12 🔗 Morbus has quit IRC (Quit: http://www.disobey.com/)
18:15 🔗 Morbus has joined #archiveteam
18:19 🔗 khaoohs has quit IRC (Read error: Connection reset by peer)
18:20 🔗 khaoohs has joined #archiveteam
18:23 🔗 wvdp_ has joined #archiveteam
18:23 🔗 khaoohs_ has joined #archiveteam
18:25 🔗 khaoohs has quit IRC (Read error: Operation timed out)
18:27 🔗 khaoohs__ has joined #archiveteam
18:28 🔗 khaoohs_ has quit IRC (Read error: Operation timed out)
18:28 🔗 wvdp___ has quit IRC (Read error: Operation timed out)
18:51 🔗 primus104 has joined #archiveteam
18:52 🔗 dashcloud has quit IRC (Quit: No Ping reply in 180 seconds.)
18:54 🔗 dashcloud has joined #archiveteam
18:56 🔗 dashcloud has quit IRC (Read error: Operation timed out)
18:58 🔗 dashcloud has joined #archiveteam
19:19 🔗 RichardG has quit IRC (Remote host closed the connection)
20:09 🔗 wvdp_ has quit IRC (Read error: Operation timed out)
20:11 🔗 beardicus has quit IRC (Ping timeout: 483 seconds)
20:32 🔗 RichardG has joined #archiveteam
20:45 🔗 joepie91 I'm sorry, User. I'm afraid I can't do that.
20:45 🔗 joepie91 Please limit --concurrent to 20 or lower to avoid exhausting resources or triggering bugs.
20:45 🔗 joepie91 @ db48x arkiver
20:45 🔗 joepie91 tried to run it at lots
20:45 🔗 joepie91 :p
20:45 🔗 habi has joined #archiveteam
20:46 🔗 habi has left
20:48 🔗 schbirid pipeline is the gnome3 of download scripts
20:55 🔗 joepie91 has quit IRC (Ping timeout: 252 seconds)
20:59 🔗 armitage_ has joined #archiveteam
20:59 🔗 armitage_ has quit IRC (Client Quit)
21:00 🔗 PurpleSym has quit IRC (Remote host closed the connection)
21:01 🔗 asdf has quit IRC (http://www.mibbit.com ajax IRC Client)
21:02 🔗 aaaaaaaaa you can clone into another directory and start another pipeline
21:07 🔗 joepie91 has joined #archiveteam
21:13 🔗 beardicus has joined #archiveteam
21:33 🔗 asdf has joined #archiveteam
23:01 🔗 balrog sounds like the ashleymadison data dump is at http://yuc3i3hat65rpl7t.onion/stuff/DarkHactivism.txt or magnet:?xt=urn:btih:40ae8a90de40ca3afa763c8edb43fc1fc47d75f1&dn=dmps fwiw
23:24 🔗 db48x joepie91: yep
23:24 🔗 db48x actually in this case I was happy for the reminder
23:24 🔗 joepie91 db48x: can I convince it otherwise?
23:24 🔗 joepie91 :p
23:24 🔗 db48x forgot I was running it on a machine with a lower amount of ram than normal :P
23:25 🔗 db48x sure, edit it
23:25 🔗 aaaaaaaaa you could but seesaw gets weird at high concurrencies
23:26 🔗 db48x pip install -e is your friend here
23:26 🔗 db48x easier to just run multiple copies
23:27 🔗 joepie91 db48x: are these all the items...?
23:27 🔗 db48x these?
23:28 🔗 joepie91 db48x: in the tracker currently
23:28 🔗 db48x ah, no
23:28 🔗 db48x I don't know how many arkiver said that there were
23:29 🔗 db48x hrm, ascom is catching up
23:29 🔗 aaaaaaaaa he is usually good at having an items repo
23:30 🔗 db48x yep: https://github.com/ArchiveTeam/blingee-items/tree/master/ADDED
23:30 🔗 aaaaaaaaa of course, that one doesn't have a todo folder
23:31 🔗 garyrh for just the blingees, it's >1,300,000 items
23:31 🔗 joepie91 how much total size, est?
23:32 🔗 garyrh at least 25tb iirc
23:32 🔗 db48x well, the average is 14MB each so far
23:32 🔗 garyrh there's also 22.5M users, thousands of groups, etc.
23:32 🔗 aaaaaaaaa http://archive.fart.website/bin/irclogger_log/archiveteam?date=2015-08-16,Sun&sel=72#l68
23:32 🔗 aaaaaaaaa he estimated 130 TB there
23:33 🔗 aaaaaaaaa actually 150 TB
23:33 🔗 joepie91 mmk
23:33 🔗 garyrh we're only getting one size for each blingee.
23:33 🔗 garyrh it's huge, no matter what.
23:34 🔗 aaaaaaaaa wait, 14MB * 130 million blingees is on the order of 2 PB
23:35 🔗 garyrh each item is 100 blingees.
23:37 🔗 dashcloud balrog: is this an actual dump, or just one that someone is claiming came from Ashley Madison?
23:38 🔗 aaaaaaaaa ok, so around 20 TB, that is a bit more sane
23:39 🔗 aaaaaaaaa ok, I thought IA was going to have to pivot into the animated gif business
23:39 🔗 db48x heh
23:39 🔗 joepie91 lol
23:40 🔗 garyrh archivee.org
23:41 🔗 BlueMaxim has joined #archiveteam
23:54 🔗 Start http://www.wired.com/2015/08/happened-hackers-posted-stolen-ashley-madison-data/
23:59 🔗 asdf has quit IRC (http://www.mibbit.com ajax IRC Client)

irclogger-viewer