[04:10] The oecd has tiny urls: oe.cd [04:13] Looks like they made their shortener relatively easy to crawl though. [04:15] powered by yourls.org [08:38] Benjojo: I'm quite sure there's people using the Warrior with VMWare [09:22] pft: might as well [09:24] GLaDOS: ping [09:36] ? [09:38] GLaDOS: was just letting you know that my tinyarchive is still running http://ec2-54-204-142-70.compute-1.amazonaws.com:8080/ [09:38] ..and it hasn't crashed yet? [09:38] GLaDOS: pft and i made it do 102 jobs and it seems to still be working [09:38] wtf [09:39] GLaDOS: i was wondering do you have anything in your cron job? [09:39] Unless it's not crashing and it just gets random periods of inactivity.. [09:40] GLaDOS: does the equivalent page to http://ec2-54-204-142-70.compute-1.amazonaws.com:8080 stop responding? [09:40] apparently it does.. [09:40] yeah, that's not just inactivity ; if you do a GET for that and nothing comes back there's a problem [09:41] anyway i was curious about whether you have anything running from cron that's tinyarchive related on that box [09:43] because it could be that something starts up, and possibly locks the sqlite db or something, by accident, causing the tinyarchive tracker daemon to get jammed up [09:48] nope, nothing on cron [09:48] so how does the work periodically get collected? [09:49] collecting the tasks, deduping, etc [09:49] is that run manually by someone when they remember? [09:52] It's done manually when I remember/when it fucking works [09:52] Seriously, the export function doesn't even work [09:52] But cleanup.py does [10:06] what do you do to run export? [10:06] I forgot which script it was [10:07] but I pointed it to the port behind varnish [10:07] i just started up 5 more tinyback instances on ec2 to try to stress the tinyarchive tracker to freeze up [10:07] oh the export is run via the web? [10:10] oh i see now [10:10] /admin/cleanup [10:11] /admin/create ; /admin/fetch ; /admin/list [10:12] do you have a varnish log on a day that it froze up handy? [10:13] a day when it was working ok, but then stopped working? [10:15] also, if tinyarchive is running on the same box as varnish it looks like those admin commands would be available to anyone on the web, since it limits those requests to localhost [10:15] hehe [10:22] i suspect one of those admin tasks is causing the app to freeze up [10:22] but looking at the varnish log could help narrow the field a bit [11:07] GLaDOS: if you get a chance to pull a varnish log just ping me in here, or you can send it to me at ehs@pobox.com [11:07] GLaDOS: i'm not trying to meddle, just think the work urlteam is doing is crazy important [11:08] GLaDOS: so it would be nice to see the tinyarchive running more regularly [11:10] indeed [12:04] ugh, can't find any varnish logs [12:04] they're supposed to be in /var/log/varnish/ right? [13:45] no, it doesn't store logs [13:45] (by default) [13:45] run varnishlog to view them [13:50] oh [13:51] oh god [13:51] ran that [13:52] getting DDoSed (basically) by tinyback clients [18:22] are you on ubuntu? [18:23] i mean, is tinyarchive running on ubuntu? [18:24] if you and you installed varnish with apt-get you should be able to `sudo /etc/init.d/varnishncsa start` [18:25] which will drop apache looking logs in /var/log/varnish and rotate them [18:25] i'd be interested to see what the traffic looks like [18:26] i've been running with 6 different tinybacks for 8 hrs or so http://ec2-54-204-142-70.compute-1.amazonaws.com:8080/ [18:26] no problems yet [18:27] but i guess there are more warriors out there than that :) [18:27] s/you and you/you/ # oops [18:27] do you want me to run a couple today? [18:27] i'm getting no tasks available [18:28] oh wait no [18:28] pft: sure [18:28] it seems to be runnign [18:28] i can probably run another one, too [18:28] i started up 5 more clients on ec2 micro instances [18:28] they unfortunately all show up as edsu-1 [18:29] is there a reason you want them showing up as different names? [18:30] i was just going to run them all with the same name [18:30] GLaDOS: i'd be curious to see the logs, to get an idea of the volume [18:30] GLaDOS: and to see if people are subitting admin commands [18:30] pft: not really, other than seeing which vms did better [18:30] ok [18:30] same name is fine :) [18:31] i haven't tried any of the admin urls yet [18:31] ok [18:31] pft: i learned earlier today that the way to run the admin commands is to hit a particular url from localhost [18:31] i saw that in the log [18:31] but that anyone can probably submit those since it is behind varnish on localhost [18:32] oh, cool [18:33] yeah, i'm unclear why varnish is being usedi n this instance but i don't really do much stuff at scale in my day job, i need to do some reading [18:36] weird, my host from yesterday is getting No tasks available but my other host is chugging along [18:41] edsu: do you see my one instance getting no tasks available? [19:07] yes, i saw that too [19:07] kind of odd [19:08] must be a bug in how it allocates tasks by ip, or some kind of throttling gone too far [19:09] changed names and it stills ays no tasks [19:50] pft: yeah, it's somehow keyed to the host name i think [19:51] weirdness [19:52] have you started hackign up the code yet? [19:52] not yet [19:52] still waiting for it to fail :) [19:52] i guess this bit with it not getting tasks is a failure of some kind [19:53] maybe that's the only failure that people ever observed, but GLaDOS seemed to indicate that the tracker went unresponsive [19:53] maybe i'm tilting at windmills since i don't know what the actual problem is [19:53] well [19:53] clearly some sort of hang/freeze is what GLaDOS was experiencing so i think you are right to focus on that [20:30] If anybody can execute admin commands, maybe an irate url-shortening operator shut down the service :P [20:32] phillipsj: i was half wondering about that [20:33] would be nice to start the tracker back up again with varnish logging and see what's going on [20:33] not sure how to do that though, other than say it in here :)