| Time |
Nickname |
Message |
|
03:35
🔗
|
underscor |
soultcer: No work? |
|
03:35
🔗
|
underscor |
2012-10-11 07:34:34,585 tinyback.Tracker INFO: Initializing tracker at http://tracker.tinyarchive.org/v1/ |
|
03:35
🔗
|
underscor |
2012-10-11 07:34:34,604 tinyback.Tracker INFO: Initializing tracker at http://tracker.tinyarchive.org/v1/ |
|
03:35
🔗
|
underscor |
2012-10-11 07:34:35,347 tinyback.Tracker INFO: No tasks available |
|
03:35
🔗
|
underscor |
2012-10-11 07:34:35,659 tinyback.Tracker INFO: No tasks available |
|
05:03
🔗
|
soultcer |
underscor: There are only 3 different URL shorteners with tasks at the moment, so you may only run 3 threads |
|
05:07
🔗
|
soultcer |
If you don't finish a task for a shortener, you don't get another task for the same url shortener for 12 hours. |
|
05:07
🔗
|
soultcer |
The long timeout was left over from when I assumed tasks would be longer. I set it to 30 minutes now |
|
05:09
🔗
|
S[h]O[r]T |
does this project need much help? |
|
05:09
🔗
|
S[h]O[r]T |
i imagine its a constant thing |
|
05:10
🔗
|
soultcer |
Yeah, the limit is mostly ip-based. Turns out URL shorteners don't like millions of requests from the same IP |
|
05:11
🔗
|
soultcer |
So we limit it to 2 requests per second, which seems to work fine. |
|
05:11
🔗
|
soultcer |
Requires a low bandwidth, but having lots of warriors run it is still nice :D |
|
05:11
🔗
|
soultcer |
(You can ran it standalone too btw, just requires Python 2.x) |
|
23:35
🔗
|
flaushy |
hej, since i see alot of rate limiting in the urlstuff, is there a nice script for my nas to crawl? |
|
23:36
🔗
|
soultcer |
Does you NAS run python? |
|
23:36
🔗
|
flaushy |
yeah |
|
23:36
🔗
|
flaushy |
full debian |
|
23:37
🔗
|
flaushy |
i figured since it gets rate limited anyhow it might be a good thing to run this in parallel to webshot |
|
23:37
🔗
|
soultcer |
It requires only very little bandwidth, almost no storage and almost no CPU power. |
|
23:37
🔗
|
flaushy |
perfect :) |
|
23:38
🔗
|
soultcer |
You can either run it like webshots, using the pipeline script, or you can run it directly from git: |
|
23:38
🔗
|
soultcer |
git clone https://github.com/soult/tinyback.git |
|
23:38
🔗
|
soultcer |
cd tinyback |
|
23:38
🔗
|
soultcer |
./run.py -t http://tracker.tinyarchive.org/v1/ -s 60 -u flaushy |
|
23:39
🔗
|
soultcer |
Or, if you want to run multiple threads at once (e.g., 3): ./run.py -t http://tracker.tinyarchive.org/v1/ -s 60 -u flaushy -n 3 |
|
23:39
🔗
|
flaushy |
does it profit alot from multithreading? |
|
23:40
🔗
|
soultcer |
There are currently 4 shorteners that we are fetching from. One thread only gets one shortener at a time, so you could use 4 threads |
|
23:40
🔗
|
flaushy |
okie running :) |
|
23:40
🔗
|
flaushy |
thx |
|
23:40
🔗
|
joepie91 |
doesn;'t work on python 2.5.x does it? |
|
23:41
🔗
|
soultcer |
Stats are here btw: http://tracker.tinyarchive.org/v1/ |
|
23:41
🔗
|
soultcer |
joepie91: To be honest, I haven't tried |
|
23:41
🔗
|
joepie91 |
File "./run.py", line 9, in <module> |
|
23:41
🔗
|
joepie91 |
File "/root/tinyback/tinyback/__init__.py", line 56 |
|
23:41
🔗
|
joepie91 |
Traceback (most recent call last): |
|
23:41
🔗
|
joepie91 |
de1:~/tinyback# ./run.py -t http://tracker.tinyarchive.org/v1/ -s 60 -u joepie91 |
|
23:41
🔗
|
joepie91 |
import tinyback |
|
23:41
🔗
|
joepie91 |
^ |
|
23:41
🔗
|
joepie91 |
except exceptions.ServiceException as e: |
|
23:41
🔗
|
joepie91 |
:P |
|
23:41
🔗
|
joepie91 |
SyntaxError: invalid syntax |
|
23:41
🔗
|
joepie91 |
it seems to trip over the 'as' |
|
23:41
🔗
|
soultcer |
Well if that is the only problem, that should be fixable. |
|
23:42
🔗
|
joepie91 |
except exceptions.ServiceException, e: |
|
23:42
🔗
|
joepie91 |
not sure if that's the only problem |
|
23:42
🔗
|
joepie91 |
but that's where it breaks |
|
23:42
🔗
|
flaushy |
nooon is happily running away :) |
|
23:42
🔗
|
joepie91 |
forking time! |
|
23:42
🔗
|
soultcer |
Basically replace "except XX as e" with "except XX, e" |
|
23:43
🔗
|
flaushy |
ah crap forgot to nice it -.- |
|
23:43
🔗
|
flaushy |
stopping goes via touch STOP again? |
|
23:44
🔗
|
soultcer |
flaushy: Nah, just kill it with kill |
|
23:44
🔗
|
flaushy |
okie |
|
23:44
🔗
|
soultcer |
If you use threading, Ctrl+C alone won't work because SIGINTS only get received by the main thread, which is busy trying to join all the worker threads and thus doesn't handle the signal. (It's on my todo-list to fix this) |
|
23:44
🔗
|
flaushy |
no problem :) |
|
23:45
🔗
|
flaushy |
if i see this correctly, we need many ips for this, right? |
|
23:46
🔗
|
soultcer |
Yeah |
|
23:47
🔗
|
soultcer |
joepie91: Is it working now? |
|
23:49
🔗
|
soultcer |
joepie91: Can you run ./run_tests.py and tell me the output? |
|
23:49
🔗
|
joepie91 |
ImportError: No module named abc |
|
23:50
🔗
|
joepie91 |
pip gives no results |
|
23:50
🔗
|
soultcer |
It's abstract base class and in the stdlib |
|
23:52
🔗
|
soultcer |
It would be possible to just remove all the abc stuff, since it is mostly a verification that all the classes are set up properly (no missing properties, etc) |
|
23:53
🔗
|
joepie91 |
it's pretty new then I suppose |
|
23:53
🔗
|
joepie91 |
and, right |
|
23:54
🔗
|
joepie91 |
New in version 2.6. |
|
23:55
🔗
|
soultcer |
It's not strictly required. Removing all mentions of abc, replacing abtractproperty with property and removing the abstract decorators is all that is needed. Only the services.py file uses abc. |
|
23:55
🔗
|
joepie91 |
alright - I'll leave that to you though |
|
23:55
🔗
|
joepie91 |
may want to make it conditional |
|
23:55
🔗
|
joepie91 |
'if importing abc succeeds...' |
|
23:55
🔗
|
joepie91 |
I did submit a pull request for the exception stuff |
|
23:56
🔗
|
soultcer |
joepie91: http://hg.python.org/cpython/raw-file/f130ce67387d/Lib/abc.py |
|
23:57
🔗
|
flaushy |
okie 2 small vpses are munching away as well :) |
|
23:57
🔗
|
soultcer |
Apparently the 2.6 version of abc is backwards compatible to 2.5, simply drop this file inside the tinyback dir and try again |
|
23:57
🔗
|
soultcer |
joepie91: I assume you are ok with the GPLv3 license? |
|
23:57
🔗
|
joepie91 |
mm, shouldn't that be distributed with it then? |
|
23:57
🔗
|
joepie91 |
and, sure |
|
23:57
🔗
|
joepie91 |
my code is always licensed as WTFPL by default |
|
23:57
🔗
|
joepie91 |
which allows any kind of reuse |
|
23:57
🔗
|
joepie91 |
literally 'any' |
|
23:58
🔗
|
joepie91 |
including relicensing :) |
|
23:58
🔗
|
joepie91 |
most compatible license on earth! :P |
|
23:58
🔗
|
GitHub66 |
[tinyback/master] Change exception syntax in another place - Sven Slootweg |
|
23:58
🔗
|
GitHub66 |
[tinyback/master] Change exception syntax in yet *another* place - Sven Slootweg |
|
23:58
🔗
|
GitHub66 |
[tinyback/master] Make script work on older Python versions as well by changing exception syntax - Sven Slootweg |
|
23:58
🔗
|
GitHub66 |
[tinyback] soult pushed 4 new commits to master: https://github.com/soult/tinyback/compare/0b806cda0795...fbac1f1aef51 |