[00:30] @ERROR: max connections (200) reached -- try again later [00:30] Process RsyncUpload returned exit code 5 for Item postcrackxi.posterous.com [00:30] rsync error: error starting client-server protocol (code 5) at main.c(1524) [sender=3.0.7] [00:30] Oh noes [03:22] So the warrior is contained in https://github.com/ArchiveTeam/seesaw-kit [03:23] You can run [03:23] sudo pip install -e "git+https://github.com/ArchiveTeam/seesaw-kit.git@master#egg=seesaw" [03:23] to install the commands run-warrior and run-pipeline [03:24] Make the directories ~/data and ~/projects [03:24] Then run run-warrior --projects-dir ~/projects --data-dir ~/data --warrior-hq http://warriorhq.archiveteam.org --port 8001 [03:24] (note: best to use a screen session) [03:25] Note: in some cases, you will have to go into a project directory, and execute ./get-wget-lua.sh [03:27] yea after a quick look at the repos in github I got a vague idea of how it works: each project has its own scripts so warrior is not exactly a self-contained program :) [03:29] also if you look in the pipeline.py of any project is lists the paths that it looks at to find wget-lua [15:53] Hmmmm… After about 24 hours I am still attempting to upload a slug of formspring data. I think I get a little further along each time (due to rsync, natch), but … I never progress beyond 0% for each try. The errors I am seeing: "io timeout after 300 seconds -- exiting" and "rsync error: timeout in data send/receive (code 30) at io.c…" Also, it looks like max connections was reduced to 100? [15:53] Should I just it be or do something else? [15:55] tcv: let it be really [15:56] the server is slowly recovering [15:56] Smiley: Thanks. No worries unless you guys tell me I need to have worries. [15:56] it will honestly sort it's self eventually [15:56] Just unfortunate that the resync takes so long. It should hopefully be faster than before, once it's finished syncing. [16:00] you're referring to the resync associated with the hard drive failure? [16:02] http://teamarchive-1.us.archive.org:8088/mrtg/ [16:02] The drive sync is over. The machine is hammered. [16:02] I may turn rsync off for 24 hours just to let it catch its breath [16:02] Woooooow. [16:03] wow. [16:04] My thought is: If none of the projects are so time-sensitive right now, it might be a good idea to decline rsync for awhile. Warriors can still down URLs. [16:04] *download [16:07] Posterous is time-sensitive [16:10] I've knocked it to 10 [16:10] I guess technically, they're ALL time-sensitive. So, it would be precious time lost. But if the machine is literally not capable of accepting much as it is, the question becomes: Would disabling rsync for a certain number of hours allow for more collection once it comes back up vs. letting the server limp along? [16:11] "knocked it to 10." What's that mean? Max conn.? [16:13] I was going to ask alard to setup another tracker for a project.. maybe do it using a different server [17:11] tcv: I assume it's a reference to Spinal Tap's mockumentary where someone says that one of the amps "goes to eleven" (instead of the normal 0-10). "It's one louder". http://en.wikipedia.org/wiki/Up_to_eleven [17:12] ersi: Oh. Here I am over-thinking it. :-P [18:01] omf_: The tracker server is fine (if you still need it). [18:02] yes please [18:02] Maybe we can do that later? I'll be back soon. [18:03] sure [18:03] Here is the domain list: http://paste.archivingyoursh.it/yahifivide.avrasm [18:06] Do we have a backup server for FOS? [18:10] Yeah it is the hammer SketchCow uses to beat it back into being up. :) [18:10] and/or any people around him at the time [18:15] omf_: that's not an answer :P [18:15] soultcer: not that i know of, excluding GLaDOS(?) xTB (8?) system [18:18] ersi, On a serious note. FOS is part of some virtual environment which was how the bad drive was able to be rebuilt [18:24] Of course, it's a VM ontop of a petabox (Uh.. it could be rebuildable anyways, there's such things as RAID and backups) - but you answered "what's FOS?" rather than "do we have a spare server to upload to?" [18:47] I did a little experiment: Ubuntu packages for Wget+Lua, https://launchpad.net/~archiveteam/+archive/wget-lua [18:50] alard: neat! [18:52] ersi: Want access to the archiveteam team on Launchpad? (It's probably useful to share that a bit.) [18:59] alard: Sure [19:17] 01[13seesaw-kit01] 15alard tagged 060.0.14 at 06development: 02https://github.com/ArchiveTeam/seesaw-kit/commits/0.0.14