Time |
Nickname |
Message |
00:08
🔗
|
|
Start has quit IRC (Excess Flood) |
00:08
🔗
|
|
Start has joined #internetarchive.bak |
00:08
🔗
|
|
svchfoo1 sets mode: +o Start |
00:20
🔗
|
|
Start has quit IRC (Read error: Connection reset by peer) |
00:20
🔗
|
|
Start_ has joined #internetarchive.bak |
01:34
🔗
|
|
zottelbey has quit IRC (Remote host closed the connection) |
01:36
🔗
|
|
VADemon_ has quit IRC (Read error: Connection reset by peer) |
01:43
🔗
|
|
niyaje4 has joined #internetarchive.bak |
02:41
🔗
|
|
Start_ is now known as Start |
02:41
🔗
|
|
Start has quit IRC (Disconnected.) |
02:41
🔗
|
|
Start has joined #internetarchive.bak |
02:41
🔗
|
|
svchfoo3 sets mode: +o Start |
03:28
🔗
|
|
niyaje4 has quit IRC (Ping timeout: 600 seconds) |
04:54
🔗
|
SketchCow |
We just had some discussion about this. |
04:54
🔗
|
SketchCow |
(Even brewster's seeing it.) |
04:54
🔗
|
SketchCow |
We pull from all over the infrastructure, so this is something worth noticing. |
04:55
🔗
|
SketchCow |
When closure returns, I'd like to know if we need to get more clients. Because we seem to be stalled on downloads. |
04:55
🔗
|
SketchCow |
Could be slowness, but that time graph is pretty damning. |
04:55
🔗
|
SketchCow |
They really should be climbing, unless we're intentionally waiting at a certain point for number of clients. |
04:56
🔗
|
yipdw |
where's the IA network weathermap again? |
04:56
🔗
|
SketchCow |
https://monitor.archive.org/weathermap/weathermap.html |
04:56
🔗
|
yipdw |
ah ha |
04:57
🔗
|
yipdw |
wow, that's some outgoing |
05:08
🔗
|
|
niyaje4 has joined #internetarchive.bak |
05:09
🔗
|
SketchCow |
Yeah, we don't fuck around. |
08:59
🔗
|
|
niyaje4 has quit IRC (Ping timeout: 600 seconds) |
13:07
🔗
|
|
zottelbey has joined #internetarchive.bak |
13:58
🔗
|
db48x |
SketchCow: the graph is also giving us the number of files, rather than the number of bytes |
13:59
🔗
|
db48x |
the large vertical jumps could be people not reporting in regularly |
14:06
🔗
|
S[h]O[r]T |
ive started download of shard2 now |
14:42
🔗
|
ppiixx |
reports are hourly right? |
14:52
🔗
|
db48x |
yes |
14:53
🔗
|
db48x |
when you run iabak we start a background process that does git annex sync once per hour |
14:53
🔗
|
db48x |
but it's not really very robust; if it dies, or is killed, or the computer is rebooted, or any number of other things, then it won't be restarted until you run iabak again |
14:54
🔗
|
db48x |
it really should be a cron job |
14:54
🔗
|
db48x |
but we haven't gotten around to it yet |
15:01
🔗
|
hater |
are cronjobs available on *bsd/osx? |
15:01
🔗
|
midas |
yep |
15:03
🔗
|
hater |
even with non-root-access? |
15:03
🔗
|
midas |
yep |
15:06
🔗
|
db48x |
every user has their own crontab |
15:14
🔗
|
hater |
what happens if cronjobs are not available? |
15:17
🔗
|
db48x |
then we're in windows |
15:17
🔗
|
db48x |
and everything is a lot more work |
15:39
🔗
|
hater |
db48x: my custom LFS-OS does not have cronjob installed |
15:57
🔗
|
|
kyan has quit IRC (Quit: Leaving) |
16:31
🔗
|
db48x |
it'll be cron and crontab |
16:31
🔗
|
db48x |
cron is the daemon which runs the jobs |
16:31
🔗
|
db48x |
crontab is a helper program which lets you edit the job specifications (also called a crontab, or cron job table) correctly |
16:32
🔗
|
db48x |
the alternative it to use systemd timers |
16:32
🔗
|
db48x |
they're actually nicer in some ways |
16:32
🔗
|
db48x |
but not as widely available |
16:55
🔗
|
closure |
so, I've added a feature that will let us have a leaderboard |
16:55
🔗
|
db48x |
:D |
16:57
🔗
|
closure |
what it provides is a list of uuids of git-annex repos, and the size of the data they're storing |
17:12
🔗
|
closure |
midas: your iabak seems to be running an old version of git-annex. could you please update it |
17:13
🔗
|
closure |
also if someone here has username jdamery at home, you too |
17:13
🔗
|
closure |
also Sean Palmer, and someone running it as root |
17:15
🔗
|
Senji |
as root> tut tut |
17:15
🔗
|
yipdw |
meh if it's a vm who cares |
17:16
🔗
|
db48x |
yipdw: defense in depth |
17:16
🔗
|
db48x |
hopefully you're using selinux as well |
17:16
🔗
|
yipdw |
maybe but I find that "fuck it, it works" overrules that every time |
17:16
🔗
|
yipdw |
it's not me btw |
17:16
🔗
|
db48x |
and giving iabak it's own security label |
17:16
🔗
|
yipdw |
which is more pain than running it as its own user |
17:16
🔗
|
db48x |
and making sure the policy prevents it from doing crazy things |
17:17
🔗
|
db48x |
like connecting to any ports except ssh and http |
17:17
🔗
|
db48x |
https, that is |
17:18
🔗
|
yipdw |
it's probably also worth noting that a large percentage of the web runs on systems that you can log into with ec2-user@ and have passwordless sudo to root :P |
17:18
🔗
|
yipdw |
so far, no major disasters that we know of |
17:18
🔗
|
yipdw |
but digression etc |
17:19
🔗
|
Senji |
install-git-annex seems to think I have the latest version -- 20150409 |
17:21
🔗
|
yipdw |
chr |
17:21
🔗
|
yipdw |
oops |
17:57
🔗
|
|
Start has quit IRC (Read error: Connection reset by peer) |
17:57
🔗
|
|
Start has joined #internetarchive.bak |
17:57
🔗
|
|
svchfoo3 sets mode: +o Start |
18:23
🔗
|
closure |
http://iabak.archiveteam.org/stats/SHARD1.leaderboard |
18:23
🔗
|
closure |
oh, look, IA is leading |
18:24
🔗
|
db48x |
heh |
18:26
🔗
|
closure |
missing argument to sendStat |
18:27
🔗
|
closure |
missing argument to sendStat |
18:27
🔗
|
closure |
seen at the end of a shardstats-all run |
18:27
🔗
|
midas |
closure: seems that my pubkey is missing for shard2 |
18:27
🔗
|
db48x |
closure: run it with -x? |
18:28
🔗
|
Kazzy |
1517859797619: e3faaa97-6ec6-4dc3-85e3-0295dfcae38b -- |
18:29
🔗
|
Kazzy |
has no user/host in shard1? |
18:29
🔗
|
midas |
Aand enter! |
18:34
🔗
|
closure |
db48x: ok, it's because ALL.size is empty |
18:35
🔗
|
db48x |
shouldn't be |
18:35
🔗
|
closure |
ah, I broke it |
18:36
🔗
|
closure |
I changed the units from tb to bytes |
18:37
🔗
|
db48x |
that should be fine |
18:37
🔗
|
closure |
broke the parser, I'll fix it up |
18:37
🔗
|
db48x |
oh, the regex |
18:38
🔗
|
db48x |
what's with all the places it uses su? |
18:39
🔗
|
db48x |
closure: no need to convert it to terabytes |
18:39
🔗
|
closure |
those files are used on the html page.. |
18:39
🔗
|
db48x |
just use bytes throughout, and we'll change the graph |
18:40
🔗
|
closure |
it su's because the SHARD users own their repos |
18:43
🔗
|
db48x |
testing changes to this script keeps getting harder and harder |
18:43
🔗
|
db48x |
why does git annex info . need to su? |
18:44
🔗
|
closure |
because it might merge git-annex branch changes and write files |
18:46
🔗
|
db48x |
SI units?! |
18:46
🔗
|
closure |
it alreasy was |
18:46
🔗
|
* |
db48x gasps |
18:47
🔗
|
db48x |
is nothing sacred any more? |
18:48
🔗
|
closure |
http://source.git-annex.branchable.com/?p=source.git;a=blob;f=Utility/DataUnits.hs;h=6e40932ef5a8801479277bf469d9032e43a0b6ee;hb=HEAD |
18:49
🔗
|
db48x |
heh |
18:49
🔗
|
Senji |
How awkward would it be to change the uid I run iabak under? :-) |
18:49
🔗
|
closure |
Senji: chown -R |
18:50
🔗
|
Senji |
It won't care that the username doesn't match the one in the repo location any more? |
18:50
🔗
|
closure |
ssh might need you to say yes to a host key again |
18:50
🔗
|
closure |
no |
18:50
🔗
|
db48x |
lol @ line 102 |
18:50
🔗
|
Senji |
Cool. |
18:51
🔗
|
closure |
db48x: I'm bitter, I once bought an ISDN line and didn't understand their absurd units |
18:52
🔗
|
db48x |
ah |
18:54
🔗
|
db48x |
grrr: ./shardstats: line 70: syntax error near unexpected token `fi' |
19:16
🔗
|
db48x |
hrm |
19:17
🔗
|
db48x |
putting units in the leaderboard instead of bytes is problematic |
19:17
🔗
|
db48x |
oh, that's just me |
19:18
🔗
|
db48x |
where is shardstats getting an upgraded git-annex from? |
19:19
🔗
|
closure |
I temporarily threw it in /usr/bin |
19:20
🔗
|
db48x |
ah |
19:20
🔗
|
db48x |
must have done that after I tried it |
19:46
🔗
|
db48x |
testing these three lines took way longer than writing them |
19:47
🔗
|
db48x |
ok, lunch |
19:49
🔗
|
|
SN4T14_ has joined #internetarchive.bak |
19:53
🔗
|
|
SN4T14__ has quit IRC (Ping timeout: 306 seconds) |
19:58
🔗
|
closure |
ht damn |
20:38
🔗
|
|
zottelbey has quit IRC (Remote host closed the connection) |
20:43
🔗
|
Senji |
I seem to be getting some 403 Forbidden errors from the web |
20:49
🔗
|
closure |
yeah, those happenn |
20:49
🔗
|
closure |
http://iabak.archiveteam.org:8080/render/?width=836&height=522&_salt=1428871679.369&target=iabak.shardstats.leaderboard.*.all&title=leaderboard&lineMode=connected&hideLegend=true |
20:54
🔗
|
db48x |
I prefer http://iabak.archiveteam.org:8080/render/?width=1060&height=717&_salt=1428872058.728&target=keepLastValue%28highestAverage%28iabak.shardstats.leaderboard.*.shard2%2C10%29%29&from=-2hours |
20:55
🔗
|
closure |
any way to label the lines on the y axis instead of in the legend? |
21:02
🔗
|
db48x |
I don't believe so |
21:03
🔗
|
db48x |
heh: http://iabak.archiveteam.org:8080/render/?width=1060&height=717&_salt=1428872569.057&from=-2hours&target=legendValue%28keepLastValue%28highestAverage%28iabak.shardstats.leaderboard.*.all%2C10%29%29%2C%22last%22%29 |
22:39
🔗
|
closure |
oops, forgot to pull that leaderboard stat update |
22:39
🔗
|
closure |
oh, it's in. I wonder why no lines have moved yet |
22:40
🔗
|
S[h]O[r]T |
whats the leaderboard url? |
22:41
🔗
|
closure |
http://iabak.archiveteam.org:8080/render/?width=1060&height=717&_salt=1428872058.728&target=keepLastValue%28highestAverage%28iabak.shardstats.leaderboard.*.shard2%2C10%29%29&from=-2hours and http://iabak.archiveteam.org/stats/ALL.leaderboard |
23:02
🔗
|
|
kyan has joined #internetarchive.bak |
23:20
🔗
|
|
Start-mob has joined #internetarchive.bak |
23:24
🔗
|
|
Start-mob has quit IRC (Remote host closed the connection) |
23:34
🔗
|
|
balrog has quit IRC (Quit: Bye) |