[00:22] It will not be a petabyte. [04:55] emergency tracker maintaince in progress [04:55] what's the situation? [04:55] redis oom'ed [04:55] if you reboot it, we can maybe upgrade to more ram [04:56] let me know and I'll poke the upgrade button on linode's page thing [04:56] i thought swap was supposed to prevent this [04:56] that is, if you're going to upgrade it [04:56] hahahaha swap [04:56] oh [04:56] Linodes with configuration profiles referencing a 32 bit kernel are currently not eligible for this upgrade. For more information please see our switching kernels guide, or redeploy this Linode using a 64 bit distro. [04:57] we should schedule a few hours tracker downtime [04:57] its down anyways so... :P [04:57] I did it on another machine and it was fairly straightforward [04:57] would rather do this kind of thing while not under duress [04:58] chfoo: do you have access to the linode dashboard that the tracker is in? [04:58] i don't think so. do i need it? [04:58] actually, if you're going to do that [04:58] er wait a minute nm [04:59] I was going to suggest merging my weighted upload tracker PR in and deploying it, but the data structure is incompatible with existing trackers [04:59] chfoo: are you draining done sets? [04:59] oh yes [04:59] I see [05:00] jesus, why is that so huge [05:00] i just drained verizon [05:00] (integer) 2878339 [05:00] i think the claims for it are just too big [05:00] never mind, answered my own question [05:00] redis 127.0.0.1:6379> zcard verizon:out [05:00] yipdw@shilling:~$ redis-cli [05:01] yeah [05:01] our cruisers cannot repel firepower of that magnitude [05:01] we're gonna need a bigger box [05:01] bigger problem is that you can't load the claims page for verizon anymore [05:01] huh, node.js got killed too [05:01] even if the tracker were online [05:02] speaking of scheduling downtime, I might want to move the tracker to a different vps hosting co. [05:02] way down the priority list [05:03] haha shit [05:03] I forgot, this is Redis 2.6 [05:03] (error) ERR unknown command 'hscan' [05:03] hscan verizon:claims [05:03] so even manually releasing those claims is going to be a problem [05:03] hmmmmmm [05:04] the log drainer and updates channel are up again. i'm done what i was going to do other than bring back up nginx [05:04] I'm going to try and reduce the out/claims sets [05:05] shutting down redis, one moment [05:05] quick fix for memory fragmentation [05:06] brute force defragmentation [05:06] is there any other way to do it? [05:06] * xmc shrugs [05:06] i don't know of any malloc that is smart enough to defragment its heaps [05:06] well, jemalloc does, rarely [05:07] I guess copying GCs inherently defragment but whether or not they return the free memory to the OS is a different question [05:07] ah yeah [05:10] oh yeah [05:10] if we are going to schedule downtime, I want to get the tracker support scripts running in daemontools or runit [05:10] running a tracker without the log drainer is bad news [05:11] * xmc nods [05:11] * crypto__ pretends to undertstand all these words [05:12] scheduling downtime is hard to do when projects are so unpredictable :P [05:12] maybe we need tracker1 and tracker2 [05:12] now there's a good idea [05:12] does redis support distributed? [05:12] why don't we spin up a new tracker and administer it properly, then shift load over [05:13] chfoo: uh kinda [05:13] multiple read is fine [05:13] :| [05:13] but for multiple masters you really need redis-cluster or consistent hashing for writes etc [05:13] the latter is probably easier for us to do [05:13] especially since the Ruby Redis client already supports it [05:14] consistent hashing for sharding that is [05:14] it'll require changes to the tracker, but I'm already working in there anyway [05:14] tracker could really use a good dose of redis-namespace [05:15] when would you guys like another machine by? [05:15] -1 hour [05:15] how about tomorrow [05:15] no seriously any time is fine [05:15] :P [05:15] i still have the puush tracker running if it's needed [05:15] I recently joined a xen collective, so I can provision machines more cheaply [05:15] how much power does that have? [05:18] shit, I wonder if I can even HKEYS the verizon claims hash [05:18] you know, at this point, I think I should just delete verizon:claims and just add everything back to verizon:todo [05:18] comments? [05:19] would it be possible to create a new project under the tracker and shove the claims to that one? [05:19] yeah, but I'd still be unable to read the claims [05:19] there's just too much dataw [05:20] read them via the tracker UI that is [05:20] I do have all the keys in the hash, so I can start trimming them that wa [05:20] y [05:20] oh, well, i don't really know. you should ask Arkiver2 [05:20] ok [05:20] Arkiver2: ^ [05:21] if we told antirez what we're doing with redis he'd probably say that we're fucking idiots [05:21] heh [05:22] lol [05:23] if anyone doesn't know, he personally contacted me to fix a data losing bug when i got mad at redis [05:25] oh? [05:28] i didn't read the faq about the overcommit setting and for days redis didn't do a dump of the db and then it oom'ed. [05:30] ahh [05:33] http://osdir.com/ml/Redis/2013-08/msg00292.html [05:33] anyway, i won't be touching the tracker. i have to sleep now [05:35] 'night [05:36] hm [05:37] ok [05:42] so I just released all verizon claims <= 2014-09-04 00:00:00Z [05:42] hopefully this makes things not suck [07:47] Boop [07:49] Bop [07:49] brap [14:34] any update on tracker status? [16:58] http://assets.www.npo.nl/uploads/media_item/media_item/62/66/Digitaal_geheugen_verlies_3_cover-1409907995.jpg [16:58] This evening on NPO2 in the Netherlands at 21:05 (Dutch time) SketchCow will be online in a documentary [16:59] see the preview here: http://www.npo.nl/vpro-tegenlicht/07-09-2014/VPWON_1209790 [17:12] is there a well-working way to archive single pieces content from facebook (statuses or images with comments)? [17:15] Arkiver2: is that a sombrero hat in SketchCow room? [17:16] godane: hah, yep! [17:59] midas joepie91: don't forget to watch SketchCow this evening! NPO2 21:05 [18:19] I just watched it. [18:19] (They sent me a vimeo link) [18:19] It's good. I wish they hadn't shown the front of my house. [18:20] SketchCow: I still have to watch it, excited :) [18:22] typical euros, care about their own privacy but not others' [18:26] SketchCow: i figured a way to archive videos from npo.nl [18:26] it maybe use full for your video if anything else [18:36] is there a english version of npo.nl site [18:36] without google translater [18:36] i only ask cause they are put up videos with english audio [18:47] someone with admin rights on the wiki here? :/ [18:49] the problem is: I can't change my password, because I don't know it anymore. I still have a the session cookies to "authorize" but they will expire soon. [18:54] godane: I'll try to get subtitles [19:10] looks like in mplayer ts from npo moves too fast [19:10] xine plays them fine [19:18] godane: version? [19:23] ? [19:23] mplayer is 1.1 [19:24] I swung my satellite dish around but sadly NPO2 is encrypted. Buh. [19:26] maybe try mplayer2 [19:33] they blocked DO AMS ips :( [19:33] A question for y'all. [19:34] A mirror site for fanfiction.net has been noticed. Do y'all have anything to do with it? [19:38] Tikatu: link? [19:38] http://www.bluebuffalodog.com/ [19:39] nah [19:39] if it's archiveteam, it'd say archiveteam [19:39] Didn't think so but wanted to be sure. Thanks heaps! [19:39] Now I'm off to notify Xing Li that he's been pirated. [19:40] who? [19:40] Again. [19:40] oh [19:40] ff.n guy [19:40] pluesch: edit the cookies expiration date: ) [19:40] he doesn't own that shit [19:40] The owner of fanfic.net [19:40] he owns the brand and the site [19:40] not the stories :] [19:40] Exactly [19:40] I' [19:40] and it doesn't look like any of that shit got pulled over [19:41] *shrug* [19:41] here at archive team we believe the more mirrors the better [19:41] ^ [19:41] schbirid: that will work? [19:41] I'm glad y'all archived as much as you did; there's a lot of plagiarists who are being axed because they're being reported to the authors' publicists. [19:41] pluesch: i bet it does [19:42] If JK Rowling suddenly said she wasn't okay with fanfiction anymore, there goes half the site. [19:42] schbirid: okay. thank you :3 [19:44] Later! :) [20:10] meh ... tumblr api TOS forbids archiving -.- [20:12] SketchCow: you in high res: http://images.poms.omroep.nl/image/s2000/506379.png [20:12] so maybe able to archive a ton of high res screenshots that way of npo [20:15] godane, any project currently running? [20:20] got to channel #quitpic [20:21] what is this project for? [20:21] we are trying to save twipic [20:21] *twitpic [20:21] ah [20:21] http://archiveteam.org/index.php?title=TwitPic [20:26] Don't forget #verizoff, Verizon Personal Web space archiving project [20:32] SketchCow: that no downloading if not 'login bug' is stage6 now [20:32] don't know if that intended or a bug [21:24] That documentary is great [23:32] WHAT FORSOOTH, PRITHEE TELL ME THE SECRET WORD [23:35] yahoosucks [23:36] BlueMaxim: thanks (or art thou a bot?) [23:36] not a bot, just one of the people who thinks yahoo sucks :P