[07:01] soultcer: hi :-) i was looking at your tinyarchive/tinyback code and was wondering at the intended workflow of the admin scripts for building a release [07:03] soultcer: also, looking at the problem the tracker is having right now, lots of sporadic database locked errors, and thinking about switching it over to postgres instead [07:04] soultcer: luckily that appears to be a relatively easy change given the way you used web.py [07:04] soultcer: little matter of porting the sqlite database to postgres ... [07:07] soultcer: and ... just wanted to say how it was nice code to read :) [08:51] pft: if you are curious about postgres changes i made https://github.com/edsu/tinyarchive/commit/1f6e95ed60b0655d01d38e7c49fd99b24d0f6bb1 [08:52] pft: if you (or anyone else) want to help me kick the tires on it, point your tinyback at http://ec2-54-204-142-70.compute-1.amazonaws.com:8080/ [09:16] opened an issue ticket to track the work too https://github.com/ArchiveTeam/tinyarchive/issues/2 [10:31] edsu: btw, you now have commit access to the urlteam repos [10:34] thanks! [10:43] still need to figure out how to generate a release [15:20] edsu: To be honest, I think the tracker code is horrible. The whole tracker + sqlite thing was more of a test implementation and then it worked "okay enough" to let it be [15:20] About the release process: There are two separate parts to the tinyarchive project: The tracker, and the main database [15:21] The tracker hands out tasks and temporarily stores their results. The main database holds all the shortlinks. Periodically the finished tasks from the tracker are removed, and imported into the main db [15:22] The advantage is that you can have a well-connected server with little storage run the tracker, while the main db previously was stored on my home server (unstable internet connection, but way better hardware/storage) [15:23] The main database is a collection of berkeleydb databases, one database for each "service" (= url shortener). The implementation for that is in the tinyarchive/database.py file. [15:24] If you want to create your own db, you'll need to import the last release from http://urlte.am/ first. Download it, then use release_import.py to import it. The DB needs about 500 GB if I remember correctly. [15:25] Then you can import tasks using import.py. [15:25] And every 6 months or so a new release is created using create_release.py [15:27] To make it easier to get a new release via BitTorrent, it is a good to have unchanged files if no shortcode has changed. Because the files are also xz-compressed, this requires some tricks. [15:28] create_release.py compares the newly created files to the files from the previous release (--old-release= switch). If the uncompressed versions match, it just copies the compressed file from the old release instead of recompressing it. [15:28] This saves time (xz compression is slow) and allows BitTorrent to recognize the file as unchanged. [15:29] Okay, now some more things about the tracker: [15:30] I saw a mention in scrollback about allowing "admin rights" for connections from 127.0.0.1: web.py understands the x-forwarded-for header and it is crucial that whatever proxy runs in front of the tracker sends this header. [15:31] a) Because otherwise every user could create/retrieve/delete tasks and b) because the tracker only hands out 1 task per service per IP address [15:31] So if the tracker only sees 127.0.0.1 as IP address, it will not hand out tasks correctly. [15:32] Also, there should only be one thread running for the tracker. I used nginx to make sure that only 1 connection was forwarded at a time. [15:33] If there is more than 1 connection, it should _not_ corrupt the database, but it might lead to the same task being handed out multiple times, and then one user will not be able to submit his finished task. So it won't break anything, but cause needless work to be done. [15:35] Now, since you have ported the tracker to postgres, you could use database locking to ensure no tasks are handed out multiple times. Or you could improve the database schema so that handed out tasks are stored in a separate table. [15:35] Then multiple web.py threads could run at the same time and locking will only be done for a short time, during handing out tasks [16:59] this is good stuff [17:10] edsu: testing engaged [17:10] edsu: if things get out of hand and you need me to shut down the testing just send me a msg, those forward to my phone [21:33] pft: thank you sir [21:34] pft: seems to be doing ok http://ec2-54-204-142-70.compute-1.amazonaws.com:8080/ [21:34] there were just a few little wrinkles to getting the sql dump loaded into postgres [21:34] i was contemplating changing the timestamp colums to datetimes [21:34] but that would involve some more work [21:35] maybe that could be an incremental improvement for down the road? [21:35] columns [21:35] bauruine: thanks for helping out w/ the testing too :) [21:37] edsu, np [21:41] yeah, datetimes might be better [21:41] might be better gains to be found by doing some of what soultcer said [21:50] * edsu looks at scrollback [21:58] lots of good info [21:58] so i'm not testing behind varnish, which is a big difference [22:01] i see, the varnish config does forward the ip, which is good :) [22:04] soultcer: you have any suspicions about the locking errors from sqlite? https://github.com/ArchiveTeam/tinyarchive/issues/2 [22:06] That's what happens when two processes try to access the sqlite db at the same time. Ideally that should not happen. [22:07] Though it does not cause any db corruption. I think there is a config option for sqlite to wait for a couple of milliseconds for the lock instead of throwing an exception [22:08] soultcer: current the tracker seems to block so much varnish thinks it is dead [22:08] currently [22:09] If varnish is configured to only let 1 concurrent connection through, then this exception should never happen (unless varnish is somehow bypassed) [22:09] i guess the tracker itself is multithreaded, i didn't realize web.py could do that, but top definitely seems to show multiple threads [22:10] It depends on how web.py is run. webpy is just the web framework, you can run it using pretty much any threaded or non-threaded wsgi-compatible server [22:10] seems to be run without any indication of processes currently [22:10] * edsu double checks [22:11] yea, python tracker.py 127.0.0.1:7091 [22:11] soultcer: if you are curious you can take a peek yourself, i can add your pubkey [22:12] pft, baurine: i have varnish running now in front of the tracker, if you want to point your clients at it [22:12] http://ec2-54-204-142-70.compute-1.amazonaws.com/ [22:12] The easiest way is to just tell varnish to only pass 1 connection at a time through. [22:15] .max_connections [22:16] i guess we can try that, but it still might be nice to upgrade to postgresql [22:18] but perhaps it's a bit more work than i suspected, with the task creation bits [22:23] well .max_connections = 1 does make the database lock errors go away :) [22:25] but it means that varnishncsa is throwing a 503 most of the time, because the queue is full [22:31] Hehe that's the next problem with sqlite [22:32] It's slow especially with that borked db layout I made [22:36] sqlite was masking a aggregate query bug that running of postgres showed [22:36] in principle the schema is pretty straightforward and fine imho [22:37] you are too hard on yourself, the code is quite nice too ; the admin workflow is a bit difficult to grok, but i've seen so much worse :-)