#urlteam 2012-10-11,Thu

↑back Search

Time Nickname Message
03:35 🔗 underscor soultcer: No work?
03:35 🔗 underscor 2012-10-11 07:34:34,585 tinyback.Tracker INFO: Initializing tracker at http://tracker.tinyarchive.org/v1/
03:35 🔗 underscor 2012-10-11 07:34:34,604 tinyback.Tracker INFO: Initializing tracker at http://tracker.tinyarchive.org/v1/
03:35 🔗 underscor 2012-10-11 07:34:35,347 tinyback.Tracker INFO: No tasks available
03:35 🔗 underscor 2012-10-11 07:34:35,659 tinyback.Tracker INFO: No tasks available
05:03 🔗 soultcer underscor: There are only 3 different URL shorteners with tasks at the moment, so you may only run 3 threads
05:07 🔗 soultcer If you don't finish a task for a shortener, you don't get another task for the same url shortener for 12 hours.
05:07 🔗 soultcer The long timeout was left over from when I assumed tasks would be longer. I set it to 30 minutes now
05:09 🔗 S[h]O[r]T does this project need much help?
05:09 🔗 S[h]O[r]T i imagine its a constant thing
05:10 🔗 soultcer Yeah, the limit is mostly ip-based. Turns out URL shorteners don't like millions of requests from the same IP
05:11 🔗 soultcer So we limit it to 2 requests per second, which seems to work fine.
05:11 🔗 soultcer Requires a low bandwidth, but having lots of warriors run it is still nice :D
05:11 🔗 soultcer (You can ran it standalone too btw, just requires Python 2.x)
23:35 🔗 flaushy hej, since i see alot of rate limiting in the urlstuff, is there a nice script for my nas to crawl?
23:36 🔗 soultcer Does you NAS run python?
23:36 🔗 flaushy yeah
23:36 🔗 flaushy full debian
23:37 🔗 flaushy i figured since it gets rate limited anyhow it might be a good thing to run this in parallel to webshot
23:37 🔗 soultcer It requires only very little bandwidth, almost no storage and almost no CPU power.
23:37 🔗 flaushy perfect :)
23:38 🔗 soultcer You can either run it like webshots, using the pipeline script, or you can run it directly from git:
23:38 🔗 soultcer git clone https://github.com/soult/tinyback.git
23:38 🔗 soultcer cd tinyback
23:38 🔗 soultcer ./run.py -t http://tracker.tinyarchive.org/v1/ -s 60 -u flaushy
23:39 🔗 soultcer Or, if you want to run multiple threads at once (e.g., 3): ./run.py -t http://tracker.tinyarchive.org/v1/ -s 60 -u flaushy -n 3
23:39 🔗 flaushy does it profit alot from multithreading?
23:40 🔗 soultcer There are currently 4 shorteners that we are fetching from. One thread only gets one shortener at a time, so you could use 4 threads
23:40 🔗 flaushy okie running :)
23:40 🔗 flaushy thx
23:40 🔗 joepie91 doesn;'t work on python 2.5.x does it?
23:41 🔗 soultcer Stats are here btw: http://tracker.tinyarchive.org/v1/
23:41 🔗 soultcer joepie91: To be honest, I haven't tried
23:41 🔗 joepie91 File "./run.py", line 9, in <module>
23:41 🔗 joepie91 File "/root/tinyback/tinyback/__init__.py", line 56
23:41 🔗 joepie91 Traceback (most recent call last):
23:41 🔗 joepie91 de1:~/tinyback# ./run.py -t http://tracker.tinyarchive.org/v1/ -s 60 -u joepie91
23:41 🔗 joepie91 import tinyback
23:41 🔗 joepie91 ^
23:41 🔗 joepie91 except exceptions.ServiceException as e:
23:41 🔗 joepie91 :P
23:41 🔗 joepie91 SyntaxError: invalid syntax
23:41 🔗 joepie91 it seems to trip over the 'as'
23:41 🔗 soultcer Well if that is the only problem, that should be fixable.
23:42 🔗 joepie91 except exceptions.ServiceException, e:
23:42 🔗 joepie91 not sure if that's the only problem
23:42 🔗 joepie91 but that's where it breaks
23:42 🔗 flaushy nooon is happily running away :)
23:42 🔗 joepie91 forking time!
23:42 🔗 soultcer Basically replace "except XX as e" with "except XX, e"
23:43 🔗 flaushy ah crap forgot to nice it -.-
23:43 🔗 flaushy stopping goes via touch STOP again?
23:44 🔗 soultcer flaushy: Nah, just kill it with kill
23:44 🔗 flaushy okie
23:44 🔗 soultcer If you use threading, Ctrl+C alone won't work because SIGINTS only get received by the main thread, which is busy trying to join all the worker threads and thus doesn't handle the signal. (It's on my todo-list to fix this)
23:44 🔗 flaushy no problem :)
23:45 🔗 flaushy if i see this correctly, we need many ips for this, right?
23:46 🔗 soultcer Yeah
23:47 🔗 soultcer joepie91: Is it working now?
23:49 🔗 soultcer joepie91: Can you run ./run_tests.py and tell me the output?
23:49 🔗 joepie91 ImportError: No module named abc
23:50 🔗 joepie91 pip gives no results
23:50 🔗 soultcer It's abstract base class and in the stdlib
23:52 🔗 soultcer It would be possible to just remove all the abc stuff, since it is mostly a verification that all the classes are set up properly (no missing properties, etc)
23:53 🔗 joepie91 it's pretty new then I suppose
23:53 🔗 joepie91 and, right
23:54 🔗 joepie91 New in version 2.6.
23:55 🔗 soultcer It's not strictly required. Removing all mentions of abc, replacing abtractproperty with property and removing the abstract decorators is all that is needed. Only the services.py file uses abc.
23:55 🔗 joepie91 alright - I'll leave that to you though
23:55 🔗 joepie91 may want to make it conditional
23:55 🔗 joepie91 'if importing abc succeeds...'
23:55 🔗 joepie91 I did submit a pull request for the exception stuff
23:56 🔗 soultcer joepie91: http://hg.python.org/cpython/raw-file/f130ce67387d/Lib/abc.py
23:57 🔗 flaushy okie 2 small vpses are munching away as well :)
23:57 🔗 soultcer Apparently the 2.6 version of abc is backwards compatible to 2.5, simply drop this file inside the tinyback dir and try again
23:57 🔗 soultcer joepie91: I assume you are ok with the GPLv3 license?
23:57 🔗 joepie91 mm, shouldn't that be distributed with it then?
23:57 🔗 joepie91 and, sure
23:57 🔗 joepie91 my code is always licensed as WTFPL by default
23:57 🔗 joepie91 which allows any kind of reuse
23:57 🔗 joepie91 literally 'any'
23:58 🔗 joepie91 including relicensing :)
23:58 🔗 joepie91 most compatible license on earth! :P
23:58 🔗 GitHub66 [tinyback/master] Change exception syntax in another place - Sven Slootweg
23:58 🔗 GitHub66 [tinyback/master] Change exception syntax in yet *another* place - Sven Slootweg
23:58 🔗 GitHub66 [tinyback/master] Make script work on older Python versions as well by changing exception syntax - Sven Slootweg
23:58 🔗 GitHub66 [tinyback] soult pushed 4 new commits to master: https://github.com/soult/tinyback/compare/0b806cda0795...fbac1f1aef51

irclogger-viewer