| Time |
Nickname |
Message |
|
00:01
🔗
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
00:05
🔗
|
|
dashcloud has joined #archiveteam |
|
00:14
🔗
|
|
bentpins has joined #archiveteam |
|
00:25
🔗
|
|
philpem has quit IRC (Ping timeout: 252 seconds) |
|
00:29
🔗
|
|
Start has joined #archiveteam |
|
00:49
🔗
|
|
wvdp___ has quit IRC (Read error: Operation timed out) |
|
01:06
🔗
|
|
bentpins has quit IRC (Ping timeout: 483 seconds) |
|
01:08
🔗
|
|
godane has quit IRC (Quit: Leaving.) |
|
01:08
🔗
|
|
godane has joined #archiveteam |
|
01:09
🔗
|
|
JesseW has joined #archiveteam |
|
01:14
🔗
|
|
primus104 has quit IRC (Leaving.) |
|
01:43
🔗
|
|
signius has quit IRC (Read error: Operation timed out) |
|
01:49
🔗
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
01:50
🔗
|
|
pfallenop has quit IRC (Ping timeout: 606 seconds) |
|
01:53
🔗
|
|
dashcloud has joined #archiveteam |
|
01:56
🔗
|
|
signius has joined #archiveteam |
|
02:26
🔗
|
|
vitzli has joined #archiveteam |
|
02:44
🔗
|
|
logan has quit IRC (Read error: Operation timed out) |
|
02:50
🔗
|
|
bentpins has joined #archiveteam |
|
03:32
🔗
|
|
xk_id has joined #archiveteam |
|
03:34
🔗
|
|
logan has joined #archiveteam |
|
03:39
🔗
|
|
xk_id has quit IRC (Remote host closed the connection) |
|
03:40
🔗
|
|
xk_id has joined #archiveteam |
|
03:57
🔗
|
|
BlueMaxim has joined #archiveteam |
|
03:59
🔗
|
|
vitzli has quit IRC (Quit: Leaving) |
|
04:05
🔗
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
04:10
🔗
|
|
dashcloud has joined #archiveteam |
|
04:11
🔗
|
|
aaaaaaaaa has quit IRC (Leaving) |
|
04:16
🔗
|
|
xk_id has quit IRC (Remote host closed the connection) |
|
04:24
🔗
|
|
xk_id has joined #archiveteam |
|
04:30
🔗
|
|
xk_id has quit IRC (Ping timeout: 369 seconds) |
|
04:31
🔗
|
|
JesseW has quit IRC (Read error: Operation timed out) |
|
04:36
🔗
|
|
JesseW has joined #archiveteam |
|
04:49
🔗
|
joepie91 |
https://code.stypi.com/e9fbieg3 |
|
04:49
🔗
|
joepie91 |
code pasting site |
|
04:49
🔗
|
joepie91 |
shutting down |
|
04:49
🔗
|
joepie91 |
september 3 |
|
04:49
🔗
|
joepie91 |
8 chars lowercase alphanumeric IDs |
|
04:53
🔗
|
|
khaoohs has quit IRC (Read error: Operation timed out) |
|
04:53
🔗
|
|
JesseW has quit IRC (Leaving.) |
|
04:56
🔗
|
|
xk_id has joined #archiveteam |
|
04:58
🔗
|
|
Emcy_ has quit IRC (Ping timeout: 483 seconds) |
|
05:00
🔗
|
bentpins |
They don't seem sequential, maybe it'd be easiest just asking nicely for a dump? |
|
05:00
🔗
|
|
khaoohs has joined #archiveteam |
|
05:05
🔗
|
|
xk_id has quit IRC (Read error: Operation timed out) |
|
05:15
🔗
|
|
xk_id has joined #archiveteam |
|
05:21
🔗
|
|
Start_ has joined #archiveteam |
|
05:21
🔗
|
|
Start has quit IRC (Write error: Broken pipe) |
|
05:26
🔗
|
|
espes__ has quit IRC (Quit: leaving) |
|
05:28
🔗
|
|
espes__ has joined #archiveteam |
|
05:28
🔗
|
|
espes__ has quit IRC (Client Quit) |
|
05:28
🔗
|
|
espes__ has joined #archiveteam |
|
05:33
🔗
|
|
dashcloud has quit IRC (Quit: No Ping reply in 180 seconds.) |
|
05:34
🔗
|
|
dashcloud has joined #archiveteam |
|
05:44
🔗
|
|
Start_ is now known as Start |
|
05:49
🔗
|
|
MorbusIff has quit IRC (Read error: Operation timed out) |
|
06:02
🔗
|
|
JesseW has joined #archiveteam |
|
06:07
🔗
|
|
Start has quit IRC (Quit: Disconnected.) |
|
06:08
🔗
|
|
Start has joined #archiveteam |
|
06:22
🔗
|
|
JesseW has quit IRC (Ping timeout: 600 seconds) |
|
06:28
🔗
|
|
pfallenop has joined #archiveteam |
|
06:43
🔗
|
|
JesseW has joined #archiveteam |
|
06:51
🔗
|
|
primus104 has joined #archiveteam |
|
06:59
🔗
|
JesseW |
joepie91: added a wiki page: http://archiveteam.org/index.php?title=Stypi |
|
07:07
🔗
|
signius |
Is this BLANK for everyone else also ? https://web.archive.org/web/20120714051606/http://ftp.sun.com/ |
|
07:09
🔗
|
|
JesseW has quit IRC (Read error: Operation timed out) |
|
07:13
🔗
|
|
PurpleSym has joined #archiveteam |
|
07:51
🔗
|
dan- |
signius: redirected me to here: https://web.archive.org/web/20120714051606/http://supportfiles.sun.com/warning |
|
07:53
🔗
|
Smiley |
http://cdn.meme.am/instances/500x/57931550.jpg |
|
07:53
🔗
|
Smiley |
errrr copypaste file. |
|
07:53
🔗
|
Smiley |
Told me it was 302 and redirected. |
|
08:00
🔗
|
|
asdf has joined #archiveteam |
|
08:02
🔗
|
|
caber has joined #archiveteam |
|
08:13
🔗
|
|
vitzli has joined #archiveteam |
|
08:35
🔗
|
|
primus104 has quit IRC (Leaving.) |
|
08:42
🔗
|
|
brayden has joined #archiveteam |
|
08:42
🔗
|
|
swebb sets mode: +o brayden |
|
08:42
🔗
|
|
vitzli has quit IRC (Quit: Leaving) |
|
09:03
🔗
|
|
schbirid has joined #archiveteam |
|
09:33
🔗
|
|
Stiletto has quit IRC (Read error: Operation timed out) |
|
10:17
🔗
|
|
xk_id has quit IRC (Remote host closed the connection) |
|
10:18
🔗
|
|
xk_id has joined #archiveteam |
|
10:23
🔗
|
schbirid |
i am grabbing ftp.gameaholic.com |
|
10:28
🔗
|
arkiver |
currently we're doing around 2000 items per minute in blingee. We need to get that to 10000 items per minute if we want to finish in time. |
|
10:29
🔗
|
arkiver |
Deadline of blingee is august 25th. |
|
10:29
🔗
|
arkiver |
#tragedee |
|
10:30
🔗
|
|
xk_id has quit IRC (Read error: Operation timed out) |
|
10:30
🔗
|
Silvan |
rsync is dead... |
|
10:31
🔗
|
arkiver |
Currently for blip, yes, not for blingee |
|
10:31
🔗
|
Silvan |
ah |
|
10:31
🔗
|
Silvan |
k |
|
10:31
🔗
|
Silvan |
shall I change to blingee? |
|
10:31
🔗
|
arkiver |
for now that would be a good idea |
|
10:42
🔗
|
|
primus104 has joined #archiveteam |
|
10:51
🔗
|
|
Morbus has joined #archiveteam |
|
11:39
🔗
|
|
deathy has quit IRC (Ping timeout: 252 seconds) |
|
12:09
🔗
|
|
Wyatts has quit IRC (Remote host closed the connection) |
|
12:12
🔗
|
|
Wyatts has joined #archiveteam |
|
12:15
🔗
|
schbirid |
ftp://ftp:any@grandis.nu |
|
12:18
🔗
|
|
mafrasi2 has quit IRC (No Ping reply in 180 seconds.) |
|
12:19
🔗
|
|
mafrasi2 has joined #archiveteam |
|
12:26
🔗
|
|
deathy has joined #archiveteam |
|
12:48
🔗
|
|
deathy has quit IRC (Ping timeout: 252 seconds) |
|
13:30
🔗
|
|
primus104 has quit IRC (Leaving.) |
|
13:35
🔗
|
|
deathy has joined #archiveteam |
|
13:39
🔗
|
|
Emcy_ has joined #archiveteam |
|
13:44
🔗
|
|
brayden has quit IRC (Quit: Leaving) |
|
13:50
🔗
|
|
PurpleSym has quit IRC (Remote host closed the connection) |
|
13:56
🔗
|
|
wvdp___ has joined #archiveteam |
|
14:03
🔗
|
|
mafrasi2 has quit IRC (No Ping reply in 180 seconds.) |
|
14:03
🔗
|
|
zyphlar has quit IRC (Ping timeout: 252 seconds) |
|
14:03
🔗
|
|
russss__ has quit IRC (Ping timeout: 252 seconds) |
|
14:03
🔗
|
|
russss__ has joined #archiveteam |
|
14:03
🔗
|
|
codl_ has quit IRC (Ping timeout: 252 seconds) |
|
14:03
🔗
|
|
Ctrl-S has quit IRC (Ping timeout: 252 seconds) |
|
14:03
🔗
|
|
kevin has quit IRC (Ping timeout: 252 seconds) |
|
14:03
🔗
|
|
zyphlar has joined #archiveteam |
|
14:04
🔗
|
|
codl_ has joined #archiveteam |
|
14:04
🔗
|
joepie91 |
arkiver: what's the bottleneck for blingee? |
|
14:04
🔗
|
joepie91 |
bandwidth / requests/ IPs? |
|
14:04
🔗
|
|
Ctrl-S has joined #archiveteam |
|
14:06
🔗
|
|
_0x2A has quit IRC (Ping timeout: 506 seconds) |
|
14:06
🔗
|
|
kevin has joined #archiveteam |
|
14:07
🔗
|
|
_0x2A has joined #archiveteam |
|
14:08
🔗
|
|
altlabel has quit IRC (Ping timeout: 506 seconds) |
|
14:14
🔗
|
|
BlueMaxim has quit IRC (Quit: Leaving) |
|
14:17
🔗
|
|
altlabel has joined #archiveteam |
|
14:17
🔗
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
14:20
🔗
|
|
dashcloud has joined #archiveteam |
|
14:23
🔗
|
arkiver |
joepie91: currently bandwidth is (on our side) |
|
14:24
🔗
|
arkiver |
or well, instances/concurrent on our side |
|
14:24
🔗
|
joepie91 |
arkiver: mmk. how crazy can I make it go |
|
14:24
🔗
|
joepie91 |
ie. how many concurrent |
|
14:24
🔗
|
arkiver |
how crazy are you able to make it go? |
|
14:25
🔗
|
arkiver |
I can always limit the number of items per minute on the tracker side so I think the more concurent the better |
|
14:28
🔗
|
joepie91 |
arkiver: I have a few bored servers, one of them is on 10gbps |
|
14:28
🔗
|
joepie91 |
lol |
|
14:28
🔗
|
arkiver |
would you be able to add a few hundred concurrent? |
|
14:28
🔗
|
arkiver |
maybe start with like 200 |
|
14:32
🔗
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
14:34
🔗
|
|
mafrasi2 has joined #archiveteam |
|
14:36
🔗
|
|
dashcloud has joined #archiveteam |
|
14:37
🔗
|
joepie91 |
arkiver: can certainly try - what's the avg disk usage I should expect per concurrent thread? |
|
14:37
🔗
|
arkiver |
I have no idea, I'm not running the project at the moment |
|
14:37
🔗
|
arkiver |
some people in #tragedee might know |
|
14:42
🔗
|
db48x |
joepie91: small, <20MB |
|
14:55
🔗
|
joepie91 |
db48x: noted, thanks |
|
14:55
🔗
|
|
PurpleSym has joined #archiveteam |
|
15:32
🔗
|
|
JesseW has joined #archiveteam |
|
15:36
🔗
|
|
deathy has quit IRC (Remote host closed the connection) |
|
15:40
🔗
|
|
deathy has joined #archiveteam |
|
16:08
🔗
|
|
JesseW has quit IRC (Leaving.) |
|
16:10
🔗
|
|
desuu has joined #archiveteam |
|
16:10
🔗
|
desuu |
Hey so im interested in installing my own archivebot instance for my own use |
|
16:11
🔗
|
desuu |
So I cloned the repo and setup the backend according to the install instructions, but where can I find the dashboard? I've tried http://localhost:4567/ but it doesnt seem like anything is listening on that port |
|
16:11
🔗
|
|
Nertsy has joined #archiveteam |
|
16:12
🔗
|
desuu |
also I have no idea what the pipeline installation is for. Can anyone help me? |
|
16:18
🔗
|
joepie91 |
paging yipdw chfoo |
|
16:18
🔗
|
joepie91 |
desuu: might take a while before you get a response |
|
16:18
🔗
|
joepie91 |
:P |
|
16:19
🔗
|
desuu |
no problem |
|
16:20
🔗
|
desuu |
my connection is horrible though so just leave a message even if im not here |
|
16:20
🔗
|
desuu |
ive got a log bot so ill see it :) |
|
16:21
🔗
|
|
desuu has quit IRC (Quit: Leaving) |
|
16:30
🔗
|
DFJustin |
you might try https://github.com/ludios/grab-site which is more geared towards local use |
|
16:33
🔗
|
zhongfu |
desuu: iirc pipeline is for the archivebot workers |
|
16:33
🔗
|
zhongfu |
archivebot backend pushes jobs to the workers |
|
16:36
🔗
|
|
RichardG has joined #archiveteam |
|
16:47
🔗
|
|
Morbus has quit IRC (Ping timeout: 252 seconds) |
|
16:52
🔗
|
|
Morbus has joined #archiveteam |
|
17:13
🔗
|
|
primus104 has joined #archiveteam |
|
17:15
🔗
|
schbirid |
https://thedrumclub.bandcamp.com/album/kit-004-the-mega-bundle |
|
17:15
🔗
|
|
philpem has joined #archiveteam |
|
17:28
🔗
|
|
chfoo has quit IRC (Ping timeout: 258 seconds) |
|
17:30
🔗
|
|
Stiletto has joined #archiveteam |
|
17:42
🔗
|
|
chfoo has joined #archiveteam |
|
17:45
🔗
|
|
primus104 has quit IRC (Leaving.) |
|
17:53
🔗
|
|
aaaaaaaaa has joined #archiveteam |
|
17:53
🔗
|
|
swebb sets mode: +o aaaaaaaaa |
|
18:12
🔗
|
|
Morbus has quit IRC (Quit: http://www.disobey.com/) |
|
18:15
🔗
|
|
Morbus has joined #archiveteam |
|
18:19
🔗
|
|
khaoohs has quit IRC (Read error: Connection reset by peer) |
|
18:20
🔗
|
|
khaoohs has joined #archiveteam |
|
18:23
🔗
|
|
wvdp_ has joined #archiveteam |
|
18:23
🔗
|
|
khaoohs_ has joined #archiveteam |
|
18:25
🔗
|
|
khaoohs has quit IRC (Read error: Operation timed out) |
|
18:27
🔗
|
|
khaoohs__ has joined #archiveteam |
|
18:28
🔗
|
|
khaoohs_ has quit IRC (Read error: Operation timed out) |
|
18:28
🔗
|
|
wvdp___ has quit IRC (Read error: Operation timed out) |
|
18:51
🔗
|
|
primus104 has joined #archiveteam |
|
18:52
🔗
|
|
dashcloud has quit IRC (Quit: No Ping reply in 180 seconds.) |
|
18:54
🔗
|
|
dashcloud has joined #archiveteam |
|
18:56
🔗
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
18:58
🔗
|
|
dashcloud has joined #archiveteam |
|
19:19
🔗
|
|
RichardG has quit IRC (Remote host closed the connection) |
|
20:09
🔗
|
|
wvdp_ has quit IRC (Read error: Operation timed out) |
|
20:11
🔗
|
|
beardicus has quit IRC (Ping timeout: 483 seconds) |
|
20:32
🔗
|
|
RichardG has joined #archiveteam |
|
20:45
🔗
|
joepie91 |
I'm sorry, User. I'm afraid I can't do that. |
|
20:45
🔗
|
joepie91 |
Please limit --concurrent to 20 or lower to avoid exhausting resources or triggering bugs. |
|
20:45
🔗
|
joepie91 |
@ db48x arkiver |
|
20:45
🔗
|
joepie91 |
tried to run it at lots |
|
20:45
🔗
|
joepie91 |
:p |
|
20:45
🔗
|
|
habi has joined #archiveteam |
|
20:46
🔗
|
|
habi has left |
|
20:48
🔗
|
schbirid |
pipeline is the gnome3 of download scripts |
|
20:55
🔗
|
|
joepie91 has quit IRC (Ping timeout: 252 seconds) |
|
20:59
🔗
|
|
armitage_ has joined #archiveteam |
|
20:59
🔗
|
|
armitage_ has quit IRC (Client Quit) |
|
21:00
🔗
|
|
PurpleSym has quit IRC (Remote host closed the connection) |
|
21:01
🔗
|
|
asdf has quit IRC (http://www.mibbit.com ajax IRC Client) |
|
21:02
🔗
|
aaaaaaaaa |
you can clone into another directory and start another pipeline |
|
21:07
🔗
|
|
joepie91 has joined #archiveteam |
|
21:13
🔗
|
|
beardicus has joined #archiveteam |
|
21:33
🔗
|
|
asdf has joined #archiveteam |
|
23:01
🔗
|
balrog |
sounds like the ashleymadison data dump is at http://yuc3i3hat65rpl7t.onion/stuff/DarkHactivism.txt or magnet:?xt=urn:btih:40ae8a90de40ca3afa763c8edb43fc1fc47d75f1&dn=dmps fwiw |
|
23:24
🔗
|
db48x |
joepie91: yep |
|
23:24
🔗
|
db48x |
actually in this case I was happy for the reminder |
|
23:24
🔗
|
joepie91 |
db48x: can I convince it otherwise? |
|
23:24
🔗
|
joepie91 |
:p |
|
23:24
🔗
|
db48x |
forgot I was running it on a machine with a lower amount of ram than normal :P |
|
23:25
🔗
|
db48x |
sure, edit it |
|
23:25
🔗
|
aaaaaaaaa |
you could but seesaw gets weird at high concurrencies |
|
23:26
🔗
|
db48x |
pip install -e is your friend here |
|
23:26
🔗
|
db48x |
easier to just run multiple copies |
|
23:27
🔗
|
joepie91 |
db48x: are these all the items...? |
|
23:27
🔗
|
db48x |
these? |
|
23:28
🔗
|
joepie91 |
db48x: in the tracker currently |
|
23:28
🔗
|
db48x |
ah, no |
|
23:28
🔗
|
db48x |
I don't know how many arkiver said that there were |
|
23:29
🔗
|
db48x |
hrm, ascom is catching up |
|
23:29
🔗
|
aaaaaaaaa |
he is usually good at having an items repo |
|
23:30
🔗
|
db48x |
yep: https://github.com/ArchiveTeam/blingee-items/tree/master/ADDED |
|
23:30
🔗
|
aaaaaaaaa |
of course, that one doesn't have a todo folder |
|
23:31
🔗
|
garyrh |
for just the blingees, it's >1,300,000 items |
|
23:31
🔗
|
joepie91 |
how much total size, est? |
|
23:32
🔗
|
garyrh |
at least 25tb iirc |
|
23:32
🔗
|
db48x |
well, the average is 14MB each so far |
|
23:32
🔗
|
garyrh |
there's also 22.5M users, thousands of groups, etc. |
|
23:32
🔗
|
aaaaaaaaa |
http://archive.fart.website/bin/irclogger_log/archiveteam?date=2015-08-16,Sun&sel=72#l68 |
|
23:32
🔗
|
aaaaaaaaa |
he estimated 130 TB there |
|
23:33
🔗
|
aaaaaaaaa |
actually 150 TB |
|
23:33
🔗
|
joepie91 |
mmk |
|
23:33
🔗
|
garyrh |
we're only getting one size for each blingee. |
|
23:33
🔗
|
garyrh |
it's huge, no matter what. |
|
23:34
🔗
|
aaaaaaaaa |
wait, 14MB * 130 million blingees is on the order of 2 PB |
|
23:35
🔗
|
garyrh |
each item is 100 blingees. |
|
23:37
🔗
|
dashcloud |
balrog: is this an actual dump, or just one that someone is claiming came from Ashley Madison? |
|
23:38
🔗
|
aaaaaaaaa |
ok, so around 20 TB, that is a bit more sane |
|
23:39
🔗
|
aaaaaaaaa |
ok, I thought IA was going to have to pivot into the animated gif business |
|
23:39
🔗
|
db48x |
heh |
|
23:39
🔗
|
joepie91 |
lol |
|
23:40
🔗
|
garyrh |
archivee.org |
|
23:41
🔗
|
|
BlueMaxim has joined #archiveteam |
|
23:54
🔗
|
Start |
http://www.wired.com/2015/08/happened-hackers-posted-stolen-ashley-madison-data/ |
|
23:59
🔗
|
|
asdf has quit IRC (http://www.mibbit.com ajax IRC Client) |