#urlteam 2013-11-06,Wed

↑back Search

Time Nickname Message
04:10 🔗 phillipsj The oecd has tiny urls: oe.cd
04:13 🔗 phillipsj Looks like they made their shortener relatively easy to crawl though.
04:15 🔗 phillipsj powered by yourls.org
08:38 🔗 ersi Benjojo: I'm quite sure there's people using the Warrior with VMWare
09:22 🔗 edsu pft: might as well
09:24 🔗 edsu GLaDOS: ping
09:36 🔗 GLaDOS ?
09:38 🔗 edsu GLaDOS: was just letting you know that my tinyarchive is still running http://ec2-54-204-142-70.compute-1.amazonaws.com:8080/
09:38 🔗 GLaDOS ..and it hasn't crashed yet?
09:38 🔗 edsu GLaDOS: pft and i made it do 102 jobs and it seems to still be working
09:38 🔗 GLaDOS wtf
09:39 🔗 edsu GLaDOS: i was wondering do you have anything in your cron job?
09:39 🔗 GLaDOS Unless it's not crashing and it just gets random periods of inactivity..
09:40 🔗 edsu GLaDOS: does the equivalent page to http://ec2-54-204-142-70.compute-1.amazonaws.com:8080 stop responding?
09:40 🔗 GLaDOS apparently it does..
09:40 🔗 edsu yeah, that's not just inactivity ; if you do a GET for that and nothing comes back there's a problem
09:41 🔗 edsu anyway i was curious about whether you have anything running from cron that's tinyarchive related on that box
09:43 🔗 edsu because it could be that something starts up, and possibly locks the sqlite db or something, by accident, causing the tinyarchive tracker daemon to get jammed up
09:48 🔗 GLaDOS nope, nothing on cron
09:48 🔗 edsu so how does the work periodically get collected?
09:49 🔗 edsu collecting the tasks, deduping, etc
09:49 🔗 edsu is that run manually by someone when they remember?
09:52 🔗 GLaDOS It's done manually when I remember/when it fucking works
09:52 🔗 GLaDOS Seriously, the export function doesn't even work
09:52 🔗 GLaDOS But cleanup.py does
10:06 🔗 edsu what do you do to run export?
10:06 🔗 GLaDOS I forgot which script it was
10:07 🔗 GLaDOS but I pointed it to the port behind varnish
10:07 🔗 edsu i just started up 5 more tinyback instances on ec2 to try to stress the tinyarchive tracker to freeze up
10:07 🔗 edsu oh the export is run via the web?
10:10 🔗 edsu oh i see now
10:10 🔗 edsu /admin/cleanup
10:11 🔗 edsu /admin/create ; /admin/fetch ; /admin/list
10:12 🔗 edsu do you have a varnish log on a day that it froze up handy?
10:13 🔗 edsu a day when it was working ok, but then stopped working?
10:15 🔗 edsu also, if tinyarchive is running on the same box as varnish it looks like those admin commands would be available to anyone on the web, since it limits those requests to localhost
10:15 🔗 ersi hehe
10:22 🔗 edsu i suspect one of those admin tasks is causing the app to freeze up
10:22 🔗 edsu but looking at the varnish log could help narrow the field a bit
11:07 🔗 edsu GLaDOS: if you get a chance to pull a varnish log just ping me in here, or you can send it to me at ehs@pobox.com
11:07 🔗 edsu GLaDOS: i'm not trying to meddle, just think the work urlteam is doing is crazy important
11:08 🔗 edsu GLaDOS: so it would be nice to see the tinyarchive running more regularly
11:10 🔗 ersi indeed
12:04 🔗 GLaDOS ugh, can't find any varnish logs
12:04 🔗 GLaDOS they're supposed to be in /var/log/varnish/ right?
13:45 🔗 Cameron_D no, it doesn't store logs
13:45 🔗 Cameron_D (by default)
13:45 🔗 Cameron_D run varnishlog to view them
13:50 🔗 GLaDOS oh
13:51 🔗 GLaDOS oh god
13:51 🔗 GLaDOS ran that
13:52 🔗 GLaDOS getting DDoSed (basically) by tinyback clients
18:22 🔗 edsu are you on ubuntu?
18:23 🔗 edsu i mean, is tinyarchive running on ubuntu?
18:24 🔗 edsu if you and you installed varnish with apt-get you should be able to `sudo /etc/init.d/varnishncsa start`
18:25 🔗 edsu which will drop apache looking logs in /var/log/varnish and rotate them
18:25 🔗 edsu i'd be interested to see what the traffic looks like
18:26 🔗 edsu i've been running with 6 different tinybacks for 8 hrs or so http://ec2-54-204-142-70.compute-1.amazonaws.com:8080/
18:26 🔗 edsu no problems yet
18:27 🔗 edsu but i guess there are more warriors out there than that :)
18:27 🔗 edsu s/you and you/you/ # oops
18:27 🔗 pft do you want me to run a couple today?
18:27 🔗 pft i'm getting no tasks available
18:28 🔗 pft oh wait no
18:28 🔗 edsu pft: sure
18:28 🔗 pft it seems to be runnign
18:28 🔗 pft i can probably run another one, too
18:28 🔗 edsu i started up 5 more clients on ec2 micro instances
18:28 🔗 edsu they unfortunately all show up as edsu-1
18:29 🔗 pft is there a reason you want them showing up as different names?
18:30 🔗 pft i was just going to run them all with the same name
18:30 🔗 edsu GLaDOS: i'd be curious to see the logs, to get an idea of the volume
18:30 🔗 edsu GLaDOS: and to see if people are subitting admin commands
18:30 🔗 edsu pft: not really, other than seeing which vms did better
18:30 🔗 pft ok
18:30 🔗 edsu same name is fine :)
18:31 🔗 edsu i haven't tried any of the admin urls yet
18:31 🔗 pft ok
18:31 🔗 edsu pft: i learned earlier today that the way to run the admin commands is to hit a particular url from localhost
18:31 🔗 pft i saw that in the log
18:31 🔗 edsu but that anyone can probably submit those since it is behind varnish on localhost
18:32 🔗 edsu oh, cool
18:33 🔗 pft yeah, i'm unclear why varnish is being usedi n this instance but i don't really do much stuff at scale in my day job, i need to do some reading
18:36 🔗 pft weird, my host from yesterday is getting No tasks available but my other host is chugging along
18:41 🔗 pft edsu: do you see my one instance getting no tasks available?
19:07 🔗 edsu yes, i saw that too
19:07 🔗 edsu kind of odd
19:08 🔗 edsu must be a bug in how it allocates tasks by ip, or some kind of throttling gone too far
19:09 🔗 pft changed names and it stills ays no tasks
19:50 🔗 edsu pft: yeah, it's somehow keyed to the host name i think
19:51 🔗 pft weirdness
19:52 🔗 pft have you started hackign up the code yet?
19:52 🔗 edsu not yet
19:52 🔗 edsu still waiting for it to fail :)
19:52 🔗 edsu i guess this bit with it not getting tasks is a failure of some kind
19:53 🔗 edsu maybe that's the only failure that people ever observed, but GLaDOS seemed to indicate that the tracker went unresponsive
19:53 🔗 edsu maybe i'm tilting at windmills since i don't know what the actual problem is
19:53 🔗 pft well
19:53 🔗 pft clearly some sort of hang/freeze is what GLaDOS was experiencing so i think you are right to focus on that
20:30 🔗 phillipsj If anybody can execute admin commands, maybe an irate url-shortening operator shut down the service :P
20:32 🔗 edsu phillipsj: i was half wondering about that
20:33 🔗 edsu would be nice to start the tracker back up again with varnish logging and see what's going on
20:33 🔗 edsu not sure how to do that though, other than say it in here :)

irclogger-viewer