#internetarchive.bak 2015-04-12,Sun

↑back Search

Time Nickname Message
00:08 🔗 Start has quit IRC (Excess Flood)
00:08 🔗 Start has joined #internetarchive.bak
00:08 🔗 svchfoo1 sets mode: +o Start
00:20 🔗 Start has quit IRC (Read error: Connection reset by peer)
00:20 🔗 Start_ has joined #internetarchive.bak
01:34 🔗 zottelbey has quit IRC (Remote host closed the connection)
01:36 🔗 VADemon_ has quit IRC (Read error: Connection reset by peer)
01:43 🔗 niyaje4 has joined #internetarchive.bak
02:41 🔗 Start_ is now known as Start
02:41 🔗 Start has quit IRC (Disconnected.)
02:41 🔗 Start has joined #internetarchive.bak
02:41 🔗 svchfoo3 sets mode: +o Start
03:28 🔗 niyaje4 has quit IRC (Ping timeout: 600 seconds)
04:54 🔗 SketchCow We just had some discussion about this.
04:54 🔗 SketchCow (Even brewster's seeing it.)
04:54 🔗 SketchCow We pull from all over the infrastructure, so this is something worth noticing.
04:55 🔗 SketchCow When closure returns, I'd like to know if we need to get more clients. Because we seem to be stalled on downloads.
04:55 🔗 SketchCow Could be slowness, but that time graph is pretty damning.
04:55 🔗 SketchCow They really should be climbing, unless we're intentionally waiting at a certain point for number of clients.
04:56 🔗 yipdw where's the IA network weathermap again?
04:56 🔗 SketchCow https://monitor.archive.org/weathermap/weathermap.html
04:56 🔗 yipdw ah ha
04:57 🔗 yipdw wow, that's some outgoing
05:08 🔗 niyaje4 has joined #internetarchive.bak
05:09 🔗 SketchCow Yeah, we don't fuck around.
08:59 🔗 niyaje4 has quit IRC (Ping timeout: 600 seconds)
13:07 🔗 zottelbey has joined #internetarchive.bak
13:58 🔗 db48x SketchCow: the graph is also giving us the number of files, rather than the number of bytes
13:59 🔗 db48x the large vertical jumps could be people not reporting in regularly
14:06 🔗 S[h]O[r]T ive started download of shard2 now
14:42 🔗 ppiixx reports are hourly right?
14:52 🔗 db48x yes
14:53 🔗 db48x when you run iabak we start a background process that does git annex sync once per hour
14:53 🔗 db48x but it's not really very robust; if it dies, or is killed, or the computer is rebooted, or any number of other things, then it won't be restarted until you run iabak again
14:54 🔗 db48x it really should be a cron job
14:54 🔗 db48x but we haven't gotten around to it yet
15:01 🔗 hater are cronjobs available on *bsd/osx?
15:01 🔗 midas yep
15:03 🔗 hater even with non-root-access?
15:03 🔗 midas yep
15:06 🔗 db48x every user has their own crontab
15:14 🔗 hater what happens if cronjobs are not available?
15:17 🔗 db48x then we're in windows
15:17 🔗 db48x and everything is a lot more work
15:39 🔗 hater db48x: my custom LFS-OS does not have cronjob installed
15:57 🔗 kyan has quit IRC (Quit: Leaving)
16:31 🔗 db48x it'll be cron and crontab
16:31 🔗 db48x cron is the daemon which runs the jobs
16:31 🔗 db48x crontab is a helper program which lets you edit the job specifications (also called a crontab, or cron job table) correctly
16:32 🔗 db48x the alternative it to use systemd timers
16:32 🔗 db48x they're actually nicer in some ways
16:32 🔗 db48x but not as widely available
16:55 🔗 closure so, I've added a feature that will let us have a leaderboard
16:55 🔗 db48x :D
16:57 🔗 closure what it provides is a list of uuids of git-annex repos, and the size of the data they're storing
17:12 🔗 closure midas: your iabak seems to be running an old version of git-annex. could you please update it
17:13 🔗 closure also if someone here has username jdamery at home, you too
17:13 🔗 closure also Sean Palmer, and someone running it as root
17:15 🔗 Senji as root> tut tut
17:15 🔗 yipdw meh if it's a vm who cares
17:16 🔗 db48x yipdw: defense in depth
17:16 🔗 db48x hopefully you're using selinux as well
17:16 🔗 yipdw maybe but I find that "fuck it, it works" overrules that every time
17:16 🔗 yipdw it's not me btw
17:16 🔗 db48x and giving iabak it's own security label
17:16 🔗 yipdw which is more pain than running it as its own user
17:16 🔗 db48x and making sure the policy prevents it from doing crazy things
17:17 🔗 db48x like connecting to any ports except ssh and http
17:17 🔗 db48x https, that is
17:18 🔗 yipdw it's probably also worth noting that a large percentage of the web runs on systems that you can log into with ec2-user@ and have passwordless sudo to root :P
17:18 🔗 yipdw so far, no major disasters that we know of
17:18 🔗 yipdw but digression etc
17:19 🔗 Senji install-git-annex seems to think I have the latest version -- 20150409
17:21 🔗 yipdw chr
17:21 🔗 yipdw oops
17:57 🔗 Start has quit IRC (Read error: Connection reset by peer)
17:57 🔗 Start has joined #internetarchive.bak
17:57 🔗 svchfoo3 sets mode: +o Start
18:23 🔗 closure http://iabak.archiveteam.org/stats/SHARD1.leaderboard
18:23 🔗 closure oh, look, IA is leading
18:24 🔗 db48x heh
18:26 🔗 closure missing argument to sendStat
18:27 🔗 closure missing argument to sendStat
18:27 🔗 closure seen at the end of a shardstats-all run
18:27 🔗 midas closure: seems that my pubkey is missing for shard2
18:27 🔗 db48x closure: run it with -x?
18:28 🔗 Kazzy 1517859797619: e3faaa97-6ec6-4dc3-85e3-0295dfcae38b --
18:29 🔗 Kazzy has no user/host in shard1?
18:29 🔗 midas Aand enter!
18:34 🔗 closure db48x: ok, it's because ALL.size is empty
18:35 🔗 db48x shouldn't be
18:35 🔗 closure ah, I broke it
18:36 🔗 closure I changed the units from tb to bytes
18:37 🔗 db48x that should be fine
18:37 🔗 closure broke the parser, I'll fix it up
18:37 🔗 db48x oh, the regex
18:38 🔗 db48x what's with all the places it uses su?
18:39 🔗 db48x closure: no need to convert it to terabytes
18:39 🔗 closure those files are used on the html page..
18:39 🔗 db48x just use bytes throughout, and we'll change the graph
18:40 🔗 closure it su's because the SHARD users own their repos
18:43 🔗 db48x testing changes to this script keeps getting harder and harder
18:43 🔗 db48x why does git annex info . need to su?
18:44 🔗 closure because it might merge git-annex branch changes and write files
18:46 🔗 db48x SI units?!
18:46 🔗 closure it alreasy was
18:46 🔗 * db48x gasps
18:47 🔗 db48x is nothing sacred any more?
18:48 🔗 closure http://source.git-annex.branchable.com/?p=source.git;a=blob;f=Utility/DataUnits.hs;h=6e40932ef5a8801479277bf469d9032e43a0b6ee;hb=HEAD
18:49 🔗 db48x heh
18:49 🔗 Senji How awkward would it be to change the uid I run iabak under? :-)
18:49 🔗 closure Senji: chown -R
18:50 🔗 Senji It won't care that the username doesn't match the one in the repo location any more?
18:50 🔗 closure ssh might need you to say yes to a host key again
18:50 🔗 closure no
18:50 🔗 db48x lol @ line 102
18:50 🔗 Senji Cool.
18:51 🔗 closure db48x: I'm bitter, I once bought an ISDN line and didn't understand their absurd units
18:52 🔗 db48x ah
18:54 🔗 db48x grrr: ./shardstats: line 70: syntax error near unexpected token `fi'
19:16 🔗 db48x hrm
19:17 🔗 db48x putting units in the leaderboard instead of bytes is problematic
19:17 🔗 db48x oh, that's just me
19:18 🔗 db48x where is shardstats getting an upgraded git-annex from?
19:19 🔗 closure I temporarily threw it in /usr/bin
19:20 🔗 db48x ah
19:20 🔗 db48x must have done that after I tried it
19:46 🔗 db48x testing these three lines took way longer than writing them
19:47 🔗 db48x ok, lunch
19:49 🔗 SN4T14_ has joined #internetarchive.bak
19:53 🔗 SN4T14__ has quit IRC (Ping timeout: 306 seconds)
19:58 🔗 closure ht damn
20:38 🔗 zottelbey has quit IRC (Remote host closed the connection)
20:43 🔗 Senji I seem to be getting some 403 Forbidden errors from the web
20:49 🔗 closure yeah, those happenn
20:49 🔗 closure http://iabak.archiveteam.org:8080/render/?width=836&height=522&_salt=1428871679.369&target=iabak.shardstats.leaderboard.*.all&title=leaderboard&lineMode=connected&hideLegend=true
20:54 🔗 db48x I prefer http://iabak.archiveteam.org:8080/render/?width=1060&height=717&_salt=1428872058.728&target=keepLastValue%28highestAverage%28iabak.shardstats.leaderboard.*.shard2%2C10%29%29&from=-2hours
20:55 🔗 closure any way to label the lines on the y axis instead of in the legend?
21:02 🔗 db48x I don't believe so
21:03 🔗 db48x heh: http://iabak.archiveteam.org:8080/render/?width=1060&height=717&_salt=1428872569.057&from=-2hours&target=legendValue%28keepLastValue%28highestAverage%28iabak.shardstats.leaderboard.*.all%2C10%29%29%2C%22last%22%29
22:39 🔗 closure oops, forgot to pull that leaderboard stat update
22:39 🔗 closure oh, it's in. I wonder why no lines have moved yet
22:40 🔗 S[h]O[r]T whats the leaderboard url?
22:41 🔗 closure http://iabak.archiveteam.org:8080/render/?width=1060&height=717&_salt=1428872058.728&target=keepLastValue%28highestAverage%28iabak.shardstats.leaderboard.*.shard2%2C10%29%29&from=-2hours and http://iabak.archiveteam.org/stats/ALL.leaderboard
23:02 🔗 kyan has joined #internetarchive.bak
23:20 🔗 Start-mob has joined #internetarchive.bak
23:24 🔗 Start-mob has quit IRC (Remote host closed the connection)
23:34 🔗 balrog has quit IRC (Quit: Bye)

irclogger-viewer