| Time |
Nickname |
Message |
|
00:06
🔗
|
|
Stiletti has quit IRC (Read error: Operation timed out) |
|
00:06
🔗
|
|
Stiletti has joined #archiveteam |
|
00:16
🔗
|
omglolbah |
Ah, trickle seems quite useful |
|
00:18
🔗
|
|
Swizzle has joined #archiveteam |
|
00:20
🔗
|
|
Aranje has quit IRC (Quit: Three sheets to the wind) |
|
00:23
🔗
|
omglolbah |
Guess I just wait for a response on key then before firing up :) |
|
00:52
🔗
|
|
pizzaiolo has quit IRC (Quit: pizzaiolo) |
|
01:09
🔗
|
|
Stiletti has quit IRC (Read error: Operation timed out) |
|
01:09
🔗
|
|
Stiletti has joined #archiveteam |
|
01:22
🔗
|
|
drumstick has quit IRC (Read error: Operation timed out) |
|
01:27
🔗
|
|
nertzy has joined #archiveteam |
|
01:46
🔗
|
|
Stiletti has quit IRC (Read error: Operation timed out) |
|
01:46
🔗
|
|
Stiletti has joined #archiveteam |
|
01:46
🔗
|
|
schbirid2 has joined #archiveteam |
|
01:49
🔗
|
|
schbirid has quit IRC (Read error: Operation timed out) |
|
02:09
🔗
|
|
jrwr has quit IRC (Read error: Operation timed out) |
|
02:10
🔗
|
|
jrwr has joined #archiveteam |
|
02:15
🔗
|
|
matthusby has quit IRC (Remote host closed the connection) |
|
02:33
🔗
|
|
matthusby has joined #archiveteam |
|
02:34
🔗
|
|
Stiletti has quit IRC (Read error: Operation timed out) |
|
02:34
🔗
|
|
Stiletti has joined #archiveteam |
|
02:43
🔗
|
|
drumstick has joined #archiveteam |
|
03:00
🔗
|
|
kitties has quit IRC (Quit: Connection closed for inactivity) |
|
03:09
🔗
|
|
Stiletti has quit IRC (Read error: Operation timed out) |
|
03:09
🔗
|
|
Stiletti has joined #archiveteam |
|
03:44
🔗
|
|
qw3rty15 has joined #archiveteam |
|
03:50
🔗
|
|
qw3rty14 has quit IRC (Read error: Operation timed out) |
|
04:10
🔗
|
|
BubuAnabe has quit IRC (Ping timeout: 268 seconds) |
|
04:19
🔗
|
|
Mateon1 has quit IRC (Ping timeout: 268 seconds) |
|
04:19
🔗
|
|
Mateon1 has joined #archiveteam |
|
04:19
🔗
|
|
Stiletti has quit IRC (Read error: Operation timed out) |
|
04:20
🔗
|
|
Stiletti has joined #archiveteam |
|
04:21
🔗
|
|
matthusb_ has joined #archiveteam |
|
04:21
🔗
|
|
matthusby has quit IRC (Read error: Operation timed out) |
|
04:21
🔗
|
|
matthusb_ has quit IRC (Remote host closed the connection) |
|
04:21
🔗
|
|
matthusby has joined #archiveteam |
|
04:34
🔗
|
|
Sk1d has quit IRC (Ping timeout: 194 seconds) |
|
04:40
🔗
|
|
Sk1d has joined #archiveteam |
|
04:57
🔗
|
|
BubuAnabe has joined #archiveteam |
|
05:20
🔗
|
|
mmm has joined #archiveteam |
|
05:22
🔗
|
|
toohighto has joined #archiveteam |
|
05:42
🔗
|
|
Stiletti has quit IRC (Read error: Operation timed out) |
|
05:42
🔗
|
|
Stiletti has joined #archiveteam |
|
06:50
🔗
|
qwerty0 |
Wow, 60GB for ArchiveBot? So it doesn't just download small pieces and immediately upload them again? |
|
06:51
🔗
|
|
Stiletti has quit IRC (Read error: Operation timed out) |
|
06:51
🔗
|
|
Stiletti has joined #archiveteam |
|
06:52
🔗
|
qwerty0 |
Also, do you know a Digital Ocean block storage volume would work? |
|
06:52
🔗
|
qwerty0 |
I've been looking for a way to help out |
|
06:52
🔗
|
|
Swizzle has quit IRC (Read error: Operation timed out) |
|
06:53
🔗
|
|
matthusby has quit IRC (Remote host closed the connection) |
|
06:53
🔗
|
|
mmm has quit IRC (Quit: Page closed) |
|
07:05
🔗
|
PurpleSym |
yipdw: Are you actually accepting new archivebot pipelines? Our wiki says no: http://archiveteam.org/index.php?title=ArchiveBot#Volunteer_a_Node |
|
07:05
🔗
|
yipdw |
that is still correct |
|
07:05
🔗
|
yipdw |
the support overhead is too high |
|
07:06
🔗
|
yipdw |
qwerty0: we do download in 5 GB pieces and upload them again. the problem is when you have many processes downloading many small pieces |
|
07:06
🔗
|
yipdw |
log files are also not chunked |
|
07:06
🔗
|
qwerty0 |
ah |
|
07:06
🔗
|
qwerty0 |
but you can limit the number of processes? |
|
07:07
🔗
|
yipdw |
yes, most people don't |
|
07:07
🔗
|
yipdw |
and you still have the log file issue |
|
07:08
🔗
|
yipdw |
if you end up with a site that has a gazillion plus one URLs your log file will be quite large |
|
07:08
🔗
|
qwerty0 |
well, if DO block storage works, i think it's fine |
|
07:08
🔗
|
qwerty0 |
it's not too expensive for 60GB |
|
07:08
🔗
|
qwerty0 |
and in any case, the bigger issue is if you're not actually accepting new pipelines |
|
07:09
🔗
|
qwerty0 |
or is what Asparagir was asking was for just donations of resources for existing pipeline operators? |
|
07:09
🔗
|
yipdw |
I don't know what Asparagir asked for |
|
07:10
🔗
|
qwerty0 |
"ust a general note: ArchiveBot needs more volunteers to set up and run pipelines. We only have a few people running a few pipelines, and some of them have bugs and need restarting or are almost full, and things like that. We need more capacity. |
|
07:10
🔗
|
qwerty0 |
So if you have free credits at AWS or Digital Ocean or whatever, or are willing to pay for a small server (a 60 GB hard drive should do, don't need much computing power), this is your time to step up." |
|
07:10
🔗
|
yipdw |
I stopped adding keys for new people because each new person adds support overhead that I do not have the time to manage |
|
07:11
🔗
|
yipdw |
i.e. things like "my pipeline went offline", "please remove the key", etc |
|
07:11
🔗
|
qwerty0 |
right, sure, makes sense |
|
07:11
🔗
|
yipdw |
those are manual tasks |
|
07:11
🔗
|
yipdw |
there are some people who have access to the control system, they're also quite busy |
|
07:11
🔗
|
qwerty0 |
so you don't know what Asparagir is planning or talking about? |
|
07:11
🔗
|
qwerty0 |
just trying to determine whether I can help out and who to talk to |
|
07:12
🔗
|
yipdw |
I can examine the logs to get a better idea, I can also ask her |
|
07:13
🔗
|
yipdw |
if the registration/deregistration system was fully automated then all of the aforementioned overhead would go away. unfortunately it's a long way from that and I haven't yet gotten to the point where I can bump that to priority 1 |
|
07:13
🔗
|
qwerty0 |
right, so at the moment all new operators have to go through you? |
|
07:14
🔗
|
yipdw |
or someone who has root access to the control node |
|
07:14
🔗
|
qwerty0 |
okay, so there's others? and Asparagir is one? |
|
07:14
🔗
|
yipdw |
that set is me, FalconK, Sanqui |
|
07:15
🔗
|
yipdw |
I don't see Asparagir's SSH key in the list, I can add it if she wants me to |
|
07:15
🔗
|
qwerty0 |
okay cool, just getting it straight |
|
07:32
🔗
|
|
Stiletti has quit IRC (Read error: Operation timed out) |
|
07:32
🔗
|
|
Stiletti has joined #archiveteam |
|
07:37
🔗
|
HCross2 |
Im ready to send my key to whoever |
|
08:04
🔗
|
|
Stiletti has quit IRC (Read error: Operation timed out) |
|
08:04
🔗
|
|
Stiletti has joined #archiveteam |
|
08:11
🔗
|
|
HarryCros has quit IRC (Read error: Connection reset by peer) |
|
08:18
🔗
|
|
HarryCros has joined #archiveteam |
|
08:43
🔗
|
|
Stiletti has quit IRC (Read error: Operation timed out) |
|
08:43
🔗
|
|
Stiletti has joined #archiveteam |
|
08:46
🔗
|
|
drumstick has quit IRC (Read error: Operation timed out) |
|
08:50
🔗
|
|
drumstick has joined #archiveteam |
|
09:05
🔗
|
Nemo_bis |
What happened with http://archiveteam.org/index.php?title=PDF_2016 ? |
|
09:08
🔗
|
|
Honno has joined #archiveteam |
|
09:19
🔗
|
|
Honno_ has quit IRC (Read error: Operation timed out) |
|
09:27
🔗
|
Nemo_bis |
http://archiveteam.org/index.php?title=PDF_manuals |
|
09:39
🔗
|
qwerty0 |
So who adds most of the things in the ArchiveBot queue? Looks like it's usually really busy |
|
09:41
🔗
|
schbirid2 |
people in #archivebot |
|
09:48
🔗
|
qwerty0 |
right, of course |
|
09:54
🔗
|
JAA |
*ba dum tss* |
|
10:10
🔗
|
JAA |
qwerty0: One thing that yipdw failed to mention by the way: wpull's database can also grow to massive sizes for large jobs (think millions of URLs). You can also run into problems when there are very large files in jobs; we tried to grab betaarchive a while ago, for example, and that downloaded several preview images of Windows 10 in parallel, crashing the pipeline. |
|
10:10
🔗
|
qwerty0 |
haha oops yeah that makes sense |
|
10:11
🔗
|
qwerty0 |
just smashes through some assumptions one might make when designing ArchiveBot |
|
10:23
🔗
|
HCross2 |
Asparagir: is it still worth me keeping my pipeline up? |
|
10:27
🔗
|
qwerty0 |
HCross2: What hosting are you using? |
|
10:28
🔗
|
HCross2 |
M247, Vienna |
|
10:29
🔗
|
qwerty0 |
oh, hadn't heard of them |
|
11:00
🔗
|
|
drumstick has quit IRC (Read error: Operation timed out) |
|
11:04
🔗
|
|
Stiletti has quit IRC (Read error: Operation timed out) |
|
11:05
🔗
|
|
Stiletti has joined #archiveteam |
|
11:21
🔗
|
|
j08nY has joined #archiveteam |
|
11:22
🔗
|
|
bRick5772 has joined #archiveteam |
|
11:50
🔗
|
zino |
Starting archivebot manually seemed to barbaric, so I made a thing: https://github.com/valgrind/abot-scripts |
|
11:50
🔗
|
zino |
Untested, because I don't have an account to test with yet... |
|
11:50
🔗
|
|
fie has quit IRC (Read error: Operation timed out) |
|
11:50
🔗
|
zino |
And wrong channel... |
|
11:58
🔗
|
|
BartoCH has quit IRC (Ping timeout: 260 seconds) |
|
12:02
🔗
|
|
pizzaiolo has joined #archiveteam |
|
12:06
🔗
|
|
BartoCH has joined #archiveteam |
|
12:32
🔗
|
|
Stiletti has quit IRC (Read error: Operation timed out) |
|
12:32
🔗
|
|
Stiletti has joined #archiveteam |
|
12:44
🔗
|
atluxity |
at SHA I was unable to find arkiver or joepie91, so I did a lightningtalk about Archive Team a bit unprepared |
|
12:44
🔗
|
atluxity |
I hope I did it justice |
|
12:44
🔗
|
atluxity |
that was really at the edge of my comfort zone, so much fun |
|
12:56
🔗
|
xmc |
atluxity: talks are good! my general rule is "i should get better at talks, so don't say no when people say i should give one" |
|
12:57
🔗
|
atluxity |
:) |
|
13:22
🔗
|
|
Stiletti has quit IRC (Read error: Operation timed out) |
|
13:22
🔗
|
|
Stiletti has joined #archiveteam |
|
13:24
🔗
|
|
matthusby has joined #archiveteam |
|
13:40
🔗
|
|
matthusby has quit IRC (Remote host closed the connection) |
|
13:47
🔗
|
|
marvinw is now known as ivan |
|
14:07
🔗
|
|
BubuAnabe has quit IRC (Ping timeout: 268 seconds) |
|
14:18
🔗
|
|
cadbury_ has joined #archiveteam |
|
14:41
🔗
|
|
Stiletti has quit IRC (Read error: Operation timed out) |
|
15:27
🔗
|
|
alex___ has joined #archiveteam |
|
16:05
🔗
|
|
matthusby has joined #archiveteam |
|
16:14
🔗
|
|
matthusb_ has joined #archiveteam |
|
16:14
🔗
|
|
matthusby has quit IRC (Read error: Connection reset by peer) |
|
16:20
🔗
|
|
matthusb_ has quit IRC (Remote host closed the connection) |
|
16:31
🔗
|
|
alex___ has quit IRC (Quit: take care ye all. Have fun!) |
|
16:32
🔗
|
|
dashcloud has quit IRC (Remote host closed the connection) |
|
16:35
🔗
|
|
matthusby has joined #archiveteam |
|
16:38
🔗
|
|
dashcloud has joined #archiveteam |
|
16:39
🔗
|
|
Aranje has joined #archiveteam |
|
16:40
🔗
|
|
toohighto has quit IRC (Remote host closed the connection) |
|
16:53
🔗
|
arkiver |
atluxity: oops, pinged you in #archiveteam-bs , but looks like you aren't there |
|
16:53
🔗
|
arkiver |
where you at? |
|
16:53
🔗
|
arkiver |
(please join #archiveteam-bs ) |
|
16:53
🔗
|
arkiver |
:) |
|
17:29
🔗
|
|
Aranje has quit IRC (Ping timeout: 245 seconds) |
|
17:38
🔗
|
|
BubuAnabe has joined #archiveteam |
|
17:46
🔗
|
|
matthusby has quit IRC (Remote host closed the connection) |
|
18:20
🔗
|
|
trvz has quit IRC (Quit: ZNC 1.6.5+deb1 - http://znc.in) |
|
18:35
🔗
|
|
fie has joined #archiveteam |
|
18:50
🔗
|
|
wabu has quit IRC (Read error: Operation timed out) |
|
19:08
🔗
|
|
matthusby has joined #archiveteam |
|
19:08
🔗
|
|
matthusby has quit IRC (Remote host closed the connection) |
|
19:10
🔗
|
|
matthusby has joined #archiveteam |
|
19:12
🔗
|
|
matthusb_ has joined #archiveteam |
|
19:12
🔗
|
|
matthusby has quit IRC (Read error: Connection reset by peer) |
|
19:16
🔗
|
|
wabu has joined #archiveteam |
|
20:09
🔗
|
|
toohighto has joined #archiveteam |
|
20:11
🔗
|
|
kitties has joined #archiveteam |
|
20:16
🔗
|
|
Swizzle has joined #archiveteam |
|
20:26
🔗
|
|
Swizzle has quit IRC (Quit: Leaving) |
|
20:29
🔗
|
|
Lord_Nigh has quit IRC (Read error: Operation timed out) |
|
20:46
🔗
|
|
Aranje has joined #archiveteam |
|
20:48
🔗
|
|
Lord_Nigh has joined #archiveteam |
|
20:52
🔗
|
|
TC01 has joined #archiveteam |
|
21:01
🔗
|
|
HarryCros has quit IRC (Read error: Connection reset by peer) |
|
21:02
🔗
|
|
HarryCros has joined #archiveteam |
|
21:27
🔗
|
|
bwn has quit IRC (Ping timeout: 268 seconds) |
|
21:28
🔗
|
|
bRick5772 has quit IRC (Quit: Leaving.) |
|
21:31
🔗
|
|
bwn has joined #archiveteam |
|
21:36
🔗
|
|
Aranje has quit IRC (Quit: Three sheets to the wind) |
|
21:45
🔗
|
|
Odd0002 has quit IRC (Remote host closed the connection) |
|
22:03
🔗
|
|
matthusb_ has quit IRC (Read error: Operation timed out) |
|
22:34
🔗
|
|
drumstick has joined #archiveteam |
|
22:39
🔗
|
|
matthusby has joined #archiveteam |
|
22:43
🔗
|
|
William has joined #archiveteam |
|
22:50
🔗
|
|
JerryStie has quit IRC (Read error: Operation timed out) |
|
22:54
🔗
|
|
ZexaronS has quit IRC (Quit: Leaving) |
|
23:20
🔗
|
|
username1 has joined #archiveteam |
|
23:23
🔗
|
|
schbirid2 has quit IRC (Read error: Operation timed out) |
|
23:36
🔗
|
|
schbirid2 has joined #archiveteam |
|
23:39
🔗
|
|
username1 has quit IRC (Read error: Operation timed out) |
|
23:39
🔗
|
|
William has quit IRC () |
|
23:46
🔗
|
|
username1 has joined #archiveteam |
|
23:46
🔗
|
|
kristian_ has joined #archiveteam |
|
23:49
🔗
|
|
schbirid2 has quit IRC (Read error: Operation timed out) |
|
23:49
🔗
|
|
matthusby has quit IRC (Remote host closed the connection) |
|
23:54
🔗
|
|
schbirid2 has joined #archiveteam |
|
23:55
🔗
|
|
nmjhyu has joined #archiveteam |
|
23:55
🔗
|
|
nmjhyu has quit IRC (Client Quit) |
|
23:55
🔗
|
|
username1 has quit IRC (Read error: Operation timed out) |