| Time |
Nickname |
Message |
|
04:10
🔗
|
phillipsj |
The oecd has tiny urls: oe.cd |
|
04:13
🔗
|
phillipsj |
Looks like they made their shortener relatively easy to crawl though. |
|
04:15
🔗
|
phillipsj |
powered by yourls.org |
|
08:38
🔗
|
ersi |
Benjojo: I'm quite sure there's people using the Warrior with VMWare |
|
09:22
🔗
|
edsu |
pft: might as well |
|
09:24
🔗
|
edsu |
GLaDOS: ping |
|
09:36
🔗
|
GLaDOS |
? |
|
09:38
🔗
|
edsu |
GLaDOS: was just letting you know that my tinyarchive is still running http://ec2-54-204-142-70.compute-1.amazonaws.com:8080/ |
|
09:38
🔗
|
GLaDOS |
..and it hasn't crashed yet? |
|
09:38
🔗
|
edsu |
GLaDOS: pft and i made it do 102 jobs and it seems to still be working |
|
09:38
🔗
|
GLaDOS |
wtf |
|
09:39
🔗
|
edsu |
GLaDOS: i was wondering do you have anything in your cron job? |
|
09:39
🔗
|
GLaDOS |
Unless it's not crashing and it just gets random periods of inactivity.. |
|
09:40
🔗
|
edsu |
GLaDOS: does the equivalent page to http://ec2-54-204-142-70.compute-1.amazonaws.com:8080 stop responding? |
|
09:40
🔗
|
GLaDOS |
apparently it does.. |
|
09:40
🔗
|
edsu |
yeah, that's not just inactivity ; if you do a GET for that and nothing comes back there's a problem |
|
09:41
🔗
|
edsu |
anyway i was curious about whether you have anything running from cron that's tinyarchive related on that box |
|
09:43
🔗
|
edsu |
because it could be that something starts up, and possibly locks the sqlite db or something, by accident, causing the tinyarchive tracker daemon to get jammed up |
|
09:48
🔗
|
GLaDOS |
nope, nothing on cron |
|
09:48
🔗
|
edsu |
so how does the work periodically get collected? |
|
09:49
🔗
|
edsu |
collecting the tasks, deduping, etc |
|
09:49
🔗
|
edsu |
is that run manually by someone when they remember? |
|
09:52
🔗
|
GLaDOS |
It's done manually when I remember/when it fucking works |
|
09:52
🔗
|
GLaDOS |
Seriously, the export function doesn't even work |
|
09:52
🔗
|
GLaDOS |
But cleanup.py does |
|
10:06
🔗
|
edsu |
what do you do to run export? |
|
10:06
🔗
|
GLaDOS |
I forgot which script it was |
|
10:07
🔗
|
GLaDOS |
but I pointed it to the port behind varnish |
|
10:07
🔗
|
edsu |
i just started up 5 more tinyback instances on ec2 to try to stress the tinyarchive tracker to freeze up |
|
10:07
🔗
|
edsu |
oh the export is run via the web? |
|
10:10
🔗
|
edsu |
oh i see now |
|
10:10
🔗
|
edsu |
/admin/cleanup |
|
10:11
🔗
|
edsu |
/admin/create ; /admin/fetch ; /admin/list |
|
10:12
🔗
|
edsu |
do you have a varnish log on a day that it froze up handy? |
|
10:13
🔗
|
edsu |
a day when it was working ok, but then stopped working? |
|
10:15
🔗
|
edsu |
also, if tinyarchive is running on the same box as varnish it looks like those admin commands would be available to anyone on the web, since it limits those requests to localhost |
|
10:15
🔗
|
ersi |
hehe |
|
10:22
🔗
|
edsu |
i suspect one of those admin tasks is causing the app to freeze up |
|
10:22
🔗
|
edsu |
but looking at the varnish log could help narrow the field a bit |
|
11:07
🔗
|
edsu |
GLaDOS: if you get a chance to pull a varnish log just ping me in here, or you can send it to me at ehs@pobox.com |
|
11:07
🔗
|
edsu |
GLaDOS: i'm not trying to meddle, just think the work urlteam is doing is crazy important |
|
11:08
🔗
|
edsu |
GLaDOS: so it would be nice to see the tinyarchive running more regularly |
|
11:10
🔗
|
ersi |
indeed |
|
12:04
🔗
|
GLaDOS |
ugh, can't find any varnish logs |
|
12:04
🔗
|
GLaDOS |
they're supposed to be in /var/log/varnish/ right? |
|
13:45
🔗
|
Cameron_D |
no, it doesn't store logs |
|
13:45
🔗
|
Cameron_D |
(by default) |
|
13:45
🔗
|
Cameron_D |
run varnishlog to view them |
|
13:50
🔗
|
GLaDOS |
oh |
|
13:51
🔗
|
GLaDOS |
oh god |
|
13:51
🔗
|
GLaDOS |
ran that |
|
13:52
🔗
|
GLaDOS |
getting DDoSed (basically) by tinyback clients |
|
18:22
🔗
|
edsu |
are you on ubuntu? |
|
18:23
🔗
|
edsu |
i mean, is tinyarchive running on ubuntu? |
|
18:24
🔗
|
edsu |
if you and you installed varnish with apt-get you should be able to `sudo /etc/init.d/varnishncsa start` |
|
18:25
🔗
|
edsu |
which will drop apache looking logs in /var/log/varnish and rotate them |
|
18:25
🔗
|
edsu |
i'd be interested to see what the traffic looks like |
|
18:26
🔗
|
edsu |
i've been running with 6 different tinybacks for 8 hrs or so http://ec2-54-204-142-70.compute-1.amazonaws.com:8080/ |
|
18:26
🔗
|
edsu |
no problems yet |
|
18:27
🔗
|
edsu |
but i guess there are more warriors out there than that :) |
|
18:27
🔗
|
edsu |
s/you and you/you/ # oops |
|
18:27
🔗
|
pft |
do you want me to run a couple today? |
|
18:27
🔗
|
pft |
i'm getting no tasks available |
|
18:28
🔗
|
pft |
oh wait no |
|
18:28
🔗
|
edsu |
pft: sure |
|
18:28
🔗
|
pft |
it seems to be runnign |
|
18:28
🔗
|
pft |
i can probably run another one, too |
|
18:28
🔗
|
edsu |
i started up 5 more clients on ec2 micro instances |
|
18:28
🔗
|
edsu |
they unfortunately all show up as edsu-1 |
|
18:29
🔗
|
pft |
is there a reason you want them showing up as different names? |
|
18:30
🔗
|
pft |
i was just going to run them all with the same name |
|
18:30
🔗
|
edsu |
GLaDOS: i'd be curious to see the logs, to get an idea of the volume |
|
18:30
🔗
|
edsu |
GLaDOS: and to see if people are subitting admin commands |
|
18:30
🔗
|
edsu |
pft: not really, other than seeing which vms did better |
|
18:30
🔗
|
pft |
ok |
|
18:30
🔗
|
edsu |
same name is fine :) |
|
18:31
🔗
|
edsu |
i haven't tried any of the admin urls yet |
|
18:31
🔗
|
pft |
ok |
|
18:31
🔗
|
edsu |
pft: i learned earlier today that the way to run the admin commands is to hit a particular url from localhost |
|
18:31
🔗
|
pft |
i saw that in the log |
|
18:31
🔗
|
edsu |
but that anyone can probably submit those since it is behind varnish on localhost |
|
18:32
🔗
|
edsu |
oh, cool |
|
18:33
🔗
|
pft |
yeah, i'm unclear why varnish is being usedi n this instance but i don't really do much stuff at scale in my day job, i need to do some reading |
|
18:36
🔗
|
pft |
weird, my host from yesterday is getting No tasks available but my other host is chugging along |
|
18:41
🔗
|
pft |
edsu: do you see my one instance getting no tasks available? |
|
19:07
🔗
|
edsu |
yes, i saw that too |
|
19:07
🔗
|
edsu |
kind of odd |
|
19:08
🔗
|
edsu |
must be a bug in how it allocates tasks by ip, or some kind of throttling gone too far |
|
19:09
🔗
|
pft |
changed names and it stills ays no tasks |
|
19:50
🔗
|
edsu |
pft: yeah, it's somehow keyed to the host name i think |
|
19:51
🔗
|
pft |
weirdness |
|
19:52
🔗
|
pft |
have you started hackign up the code yet? |
|
19:52
🔗
|
edsu |
not yet |
|
19:52
🔗
|
edsu |
still waiting for it to fail :) |
|
19:52
🔗
|
edsu |
i guess this bit with it not getting tasks is a failure of some kind |
|
19:53
🔗
|
edsu |
maybe that's the only failure that people ever observed, but GLaDOS seemed to indicate that the tracker went unresponsive |
|
19:53
🔗
|
edsu |
maybe i'm tilting at windmills since i don't know what the actual problem is |
|
19:53
🔗
|
pft |
well |
|
19:53
🔗
|
pft |
clearly some sort of hang/freeze is what GLaDOS was experiencing so i think you are right to focus on that |
|
20:30
🔗
|
phillipsj |
If anybody can execute admin commands, maybe an irate url-shortening operator shut down the service :P |
|
20:32
🔗
|
edsu |
phillipsj: i was half wondering about that |
|
20:33
🔗
|
edsu |
would be nice to start the tracker back up again with varnish logging and see what's going on |
|
20:33
🔗
|
edsu |
not sure how to do that though, other than say it in here :) |