[00:00] '2Al' is done. [00:00] that's one reason inconsolata is an awesome typeface :) [00:00] obvious differentation of 0 and O [00:00] Lucida console also works fine [00:03] I use clearlyu-clean on my terminals. [00:05] I seem to be using Monospace. I can see the difference between 0 and O, just didn't remember that I should look for a difference. :) [00:10] Is it a good idea to add everything that isn't done yet back to the queue? [00:10] I can do it in a way that should ensure that it is given to someone else than the first claimant. [00:11] how do you know what isn't done yet? [00:11] alard: whut? http://pastebin.com/raw.php?i=LAu1unPR [00:12] Well, a couple of things: I know what has been claimed but hasn't been marked done (602 items). I know what should have been done (generate a list of ids), I know what has been marked done. [00:13] ah [00:13] yea, no reason no to throw those back into the hopper [00:13] PepsiMax: A misplaced warc? [00:13] /archiveteam/anyhub-grab$ grep -l Cannot data/*/wget*.log [00:13] alard: it worked before [00:13] data/3_q/wget-3_q-1.log [00:13] data/4xH/wget-4xH-1.log [00:13] data/4xH/wget-4xH-d1.log [00:14] PepsiMax: You probably have a warc file in your data/ directory. (It looks like it, anyway.) [00:15] I do. [00:16] alard: can you see what just happend with my rsync? It looked like it started sending data you already should have... [00:17] 2011/11/18 00:16:50 [22037] rsync error: error in rsync protocol data stream (code 12) at io.c(601) [generator=3.0.7] [00:17] 2011/11/18 00:16:50 [22037] rsync error: error in rsync protocol data stream (code 12) at io.c(760) [receiver=3.0.7] [00:17] 2011/11/18 00:16:50 [22037] rsync: connection unexpectedly closed (5454 bytes received so far) [generator] [00:17] 2011/11/18 00:16:50 [22037] rsync: read error: Connection reset by peer (104) [00:17] 2011/11/18 00:16:49 [22037] data/1RE/ [00:17] 2011/11/18 00:16:50 [22037] data/1S6/ [00:18] I CTRL+c 'd [00:18] I have 130 ones that I just requeued [00:18] PepsiMax: A data/ directory has appeared. [00:18] Because they weren't done [00:18] alard: Just so you know [00:18] underscor: Well, I just added ~ 550 items back in the queue to be redone by someone, so it'll probably work. [00:19] Hmm /archiveteam/anyhub-grab/data pepsimax@8yourbox::pepsimax/anyhub/ [00:19] PepsiMax: The slashes are always tricky. You may be missing a / [00:20] this is so cofusing [00:20] I think you're uploading data to anyhub/data if you do this. [00:21] if im in the data/ dir [00:21] no [00:21] wait [00:21] The location of the main dir is [00:21] /mnt/extdisk/archiveteam/anyhub-grab/ [00:21] Can't you run the upload script? [00:21] there is? [00:22] There is, maybe I added it after you last git pulled. [00:22] i pulled [00:22] didnt saw it [00:22] who is dest? [00:22] Well, the script assumes you'll be uploading to SketchCOw. [00:22] PepsiMax: make sure your source tree contains b543af28807150554b3a0f0958615657def5df4d as an ancestor [00:23] Already up-to-date. [00:23] (or is at that commit) [00:23] what's git show HEAD --format=oneline show [00:23] yeah [00:23] but why? [00:23] then there should be an upload-finished.sh in the root of the repository [00:23] hurr [00:23] it is here [00:23] its kinda working [00:24] but i got login details of alard [00:24] i moved files around [00:24] Two options: modify the script, or ask SketchCow for an official rsync slot. [00:24] now i forgot how rsync assumes you are copying data [00:24] SketchCow: bzzzzzzzzzzzzzzzzzz [00:25] Can I haz 25GB of storage [00:27] PepsiMax: You want rsync -avP /mnt/extdisk/archiveteam/anyhub-grab/ pepsimax@8yourbox::pepsimax/anyhub/ [00:27] -avP? [00:27] aRchival, Verbose, Progress, Partial [00:28] That might give you a "group" error [00:28] if it does, you want [00:28] rsync rsync -rlptoDvP /mnt/extdisk/archiveteam/anyhub-grab/ pepsimax@8yourbox::pepsimax/anyhub/ [00:28] Er, 1 rsync [00:28] its uploading something [00:29] but god knows where [00:29] well [00:29] Oh, whoops [00:29] You want data on the end [00:29] my bad [00:29] rsync -rlptoDvP /mnt/extdisk/archiveteam/anyhub-grab/data/ pepsimax@8yourbox::pepsimax/anyhub/ [00:29] or [00:29] rsync -avP /mnt/extdisk/archiveteam/anyhub-grab/data/ pepsimax@8yourbox::pepsimax/anyhub/ [00:29] Did everyone suddenly stop their downloaders? [00:30] alard: My clients are not running [00:30] because I'm running dld-singles [00:30] Ah, I see. [00:30] It's eerily quiet on the tracker. 549 items to do, but no requesters. [00:30] :) [00:31] I'm stuck trying to fix 4z- [00:31] heh [00:31] well, h ell [00:31] yeah [00:31] I'll spin up another client [00:31] alard: i was trying to save the remaining 25GB, its nomming on 0ry now. [00:32] alard: Fired up a few clients [00:34] PepsiMax: rsync, you mean? Good. Unfortunately, I'm shutting things down for today/tonight, so you'll have to continue tomorrow. (Or see if you get hold of SketchCow, which would be even better.) [00:34] underscor; Thanks. [00:34] Well, bye all. [00:34] Good. [00:34] cya [00:35] adios [02:59] SketchCow: Batcave is about to crap itself, just so you know [05:38] I love how I can see a Splinder download completing in one window -- and then see my name be shoved off the dashboard in seconds in another window [05:45] strange, I'm not seeing anything live in the dashboard. [05:45] ah, it's an opera thing [05:49] wat [05:50] screen won't let me have more than 40 windows?!? [05:59] there's a maxwin setting you can adjust [06:00] lol, it can only be set lower than 40.. guess you'd have to recompile. what a strange thing [06:01] you have to recompile and set MAXWIN to something else [06:01] lazy goddamn c programmers [06:02] chronomex: Use tmux [06:02] It's better [06:02] :> [06:02] must save previous previous bytes in window list [06:03] pfeh, screen works fine. [06:03] for most uses. [06:03] [06:03] A Fatal Error has occurred. [06:03] Error Number: -2147205115 [06:03] Error Source: [SystemInfo: GetSystemConfig] [SystemInfo: GetSystemConfigItem] [SystemInfo: LoadSystemConfigInfo] [06:03] Welcome to Prince William County School's Parent Portal [06:03] Description: Provider: Microsoft OLE DB Provider for SQL Server Interface: IDBInitialize Description: Timeout expired [06:03] Fucking school gradebook [06:03] little Bobhy Tables must go there [06:03] to be sure, I should modify alard's excellent scraper to run a bunch of them nicely. [06:04] chronomex: well, I simply do ./dld-client.sh closure & ./dld-client.sh closure & ./dld-client.sh closure & ./dld-client.sh closure & ./dld-client.sh closure & [06:04] right. but there's a cleaner way I have in mind. [06:14] alard: Looks like jstor broke metadata scraping or something [06:14] I keep getting WARNING:root:400 POST /data (71.126.138.142): Missing argument meta [06:14] WARNING:root:400 POST /data (71.126.138.142) 3754.34ms [06:14] on my listener [06:42] * chronomex working on properly multithreadifying splinder [06:42] # reap dead children [07:19] donbex wrote some scripts [07:28] oh yeah? [07:29] I bet so, he's running so fast :P [07:38] that seems not the reason, though :-p [07:39] although part of the reason because there was a bug and at some moment he had 200 processes running [07:39] lol [07:39] this is a genius: [07:39] - Downloading blog from ----------------------.splinder.com... done, with network errors. [07:40] "is that eighteen or 22 hyphens?" [07:42] yes, a very inspiring domain name [07:45] i've put his scripts on http://toolserver.org/~nemobis/ [07:46] I'm not sure he agrees, though; ssshhh [08:05] "If I make the name totally super hard, it's gonna be secret so no one will find it" [08:06] clearly [08:12] then again, firefucks had problems loading urls with - in them earlier [08:13] on linux atleast, I think it worked on the windows version [08:15] splinder shouldn't have allowed it. [08:15] being contrary to the dns spec and all [08:30] there we go, thready version works great. [08:30] gonna put it on github and issue a pull request in a bit [08:40] wooo I own the realtime list [08:40] ish [08:51] 12/50 PID 8006 finished 'us:replicawatch7': Success. [08:51] replicawatch7, eh. excellent. [08:57] dld-streamer.sh is now in the archiveteam splinder git repository, if y'all want to use it [08:57] usage: ./dld-streamer.sh 40 or whatever number you can handle [08:58] caveats: it loads up your system right good [08:58] does it generate the same results as the old version? [08:58] (never hurts to ask) [08:58] so far as I can make it, yes. it's a modified dld-client.sh [08:59] console output from dld-single.sh goes to $username.log , which is deleted upon successful completion of the user. [09:02] multithreaded programming in bash is kind of a bitch, so I'm not completely certain if I'm properly catching failed dld-single.sh's. [09:02] I didn't even know bash did threads [09:02] a second pair of eyes would help [09:02] well, it's not threads so much as management of a lot of background tasks. [09:02] oh [09:02] so &? [09:02] yes [09:03] but then I have to save the pid I just forked off and check if it's still around periodically [09:07] chronomex: Cool. Would it be possible to make it so that it keeps running when it doesn't get a username? (With a sleep of 10 to 30 seconds in between, for example?) [09:07] I'm sure it would, but I don't have a way to test that easily. [09:09] closure just did a user named "it:MS.Dos". I wonder how that works. [09:09] oh wait, users != blogs [09:09] kind of like tumblr I suppose [09:16] oh good lord [09:16] - Parsing profile HTML to extract media urls... done. [09:16] - Downloading 834 media files... [09:16] gonna be here a while [09:16] heh [09:19] though, that's still not bad compared to the Redazione profile [09:19] I've been downloading the first blog on that profile for a week now [09:20] it would probably be a good idea for someone else could start up a download for that profile [09:20] just in case mine errors out [09:20] any idea what they're writing about? [09:21] I think it's some sort of official Splinder account [09:21] "redazione" means "editorial staff", I think [09:21] ah. [09:22] http://www.splinder.com/profile/Redazione/blogs is what worries me [09:22] journal.splinder.com is the first one out of 8 [09:23] and they all have many entries, each with many comments :P [09:25] http://www.splinder.com/myblog/comment/list/4212591/49575751?from=400 [09:25] oh, NOW I know why some of these blogs have such huge comment pages. [09:27] hahaha [09:27] lesbian strapon fisting! [09:27] wtf is that, do they have a prosthetic arm connected to a harness or something [09:28] I don't know, but I bet anime has the answer [09:28] I'm good [09:49] I once tried to archive it.wikihow.com and it failed while downloading a talk page with a 10 GiB ish history full with spam [09:50] surprisingly, deleting it didn't kill their servers [10:00] woops, dld-streamer.sh has a bug where it goes into a spinloop during stopping state [10:41] chronomex, what's the opposite of touch STOP ? [10:41] opposite how? [10:41] rm STOP i guess [10:41] I don't knw, how do you stop the stopping? [10:41] rm STOP [10:41] :) [10:41] ah ok [10:41] thanks :) [10:41] you have to do it before the script sees STOP, which it will do usually within a second or two after you create it [10:42] * Nemo_bis didn't get how it works [10:42] yep, I'm waiting for some processes to complete [10:50] it works for you too? excellent. [10:51] not that I don't expect it, but that's good to hear [10:53] no, I didn't start it yet because I'm stopping the running dld-client now [10:53] 44 left [10:53] aye [10:54] it's safe to run in a different terminal. [10:54] similarly, you can run as many dld-client.sh in that directory as you want at once [10:54] hmm, yes, I'll do so because they're so slooooooooooooow now [10:55] wait [10:55] I suggest you run it as [10:55] ionice -c 3 -n 5 nice -n 10 ./dld-streamer.sh 50 [10:56] that way it's less likely to completely eat your machine [10:58] right now it's not a problem [10:58] hrm [10:58] I tried ionice -c 3 but then many processes died because were not able to write to disk [10:59] 17.5 KB/s is not the ideal transfer rate for this 40 gigs of data [10:59] heh [11:00] especially since the machines are right next to each other [11:04] hrm [11:04] db48x: eew [11:04] one of my splinder clients is stuck retrying --------.splinder.com [11:06] it:luke1989 [11:08] chronomex: wait, you can pass eg 50 to make it dl 50 at once? [11:08] yeah [11:09] * Schbirid stops a lot of terminal windows... [11:09] heh [11:09] I suggest sticking to around 50 in each streamer instance, you can run multiple streamers at once [11:11] see how much it loads your machine ... it can really churn your disk around. [11:11] yeah [11:12] got around 12 io/s and latency of 100ms but i have no idea how much it is [11:12] just compared to the normal almost-0 it sure shows up on the graphs [11:12] lol [11:25] the download speed is going down [11:35] for me too [11:35] probably servers overloaded [11:36] (this is peak hour) [11:36] archiveteam, your free load testing service [11:37] nah, I doubt it, we're not pulling so much [11:37] <3 [11:43] chronomex, is it normal that it's always the 49th or 50th process to tell me that a user is completed? http://p.defau.lt/?WnNq7oqrkhMYTsNPAhdNvQ [11:43] oh, yeah, that deserves some explanation [11:44] the first number is how many processes are running, the second is how many you want [11:44] ah, ok [11:44] I don't number them individually, I just count how many there are [11:44] so, yes. [11:44] so only PID tells something [11:44] right [11:44] ok [11:44] yep [11:46] huh. https://plus.google.com/112313173544747389010/posts/UouzhaSbB1M [12:09] So I ahve ~15 splinder processes that are still running 12 hours after a touch STOP [12:09] and they are still downloading [12:13] chronomex: Bwahaha [12:13] Cameron_D: huge downloads then. [12:13] What a bunch of fucking retards Backupify is [12:13] PepsiMax, yeah :/ [12:14] eventually every client bumps into a big user and gets stuck with it :/ [12:19] but the time between the small ones is so big. [12:20] so you start a few clients, to speed up the small user-waiting time [12:20] and booom, 4 clients sucking the internet. [12:25] I think I'm doing it right... http://i.imgur.com/Ln10H.jpg [12:26] lol skynet [12:26] Unoriginal name is unoriginal :P [14:00] bleerg [15:11] No more usernames available. Entering reap mode... [15:11] ?? [15:11] Any chance of making the dld-client try harded or be more patient when telling the tracker it's done? I have a quarter of my processes that stopped this morning because they didn't hear back quickly enough. [15:11] s/harded/harder/ [15:24] the same for me [15:25] and dld-streamer stops everything the first time it doesn't receive a user [16:55] Summary of payments: * Your withdrawal of $107981.00 succeeded for a bank account. [16:55] TIME TO GO SHOPPPPPPINGGGGGGGG [17:05] >:) [17:05] Kickstarter is an awesome invention [17:10] what will you be shooting with SketchCow? [17:17] "./dld-client.sh mynick 30" will try 30 at once, correct? [17:27] Schbirid, that's dld-streamer.sh [17:28] oh [17:29] oh yes [17:29] thanks [17:53] For those who missed it: ARCHIVE TEAM: A Distributed Preservation of Service Attack http://youtu.be/-2ZTmuX3cog [17:56] anyhub deaded? [18:17] i found out how to make wikipedia work offline: http://users.softlab.ece.ntua.gr/~ttsiod/buildWikipediaOffline.html [18:18] its best when you need a mirror of it locally [18:20] heh heh.. best to mirror *.gr while you are at it :) [18:26] why is it that wikipedia dumps are in bz2? [18:27] i have see some are not compressed at all [18:27] part of me thinks wikipedia dumps need to be done in lzma or xz [18:29] there are also 7z versions [18:30] bzip archives are needed for some applications which process them without uncompressing them [18:30] ok [18:30] thought there was a bzip2recover for xz [18:30] or lzma [18:30] and lzma doesn't improve much in some cases; it's very useful for complete histories with a lot of duplicate text [18:31] i have seen 10% improvement [18:31] that's not much :) [18:31] 7z archives of full history are about 1/5 of bzip2 [18:31] it is when your talking 7.3gb [18:31] saves 700mb [18:32] which is not much if the archive uncompressed is 730 GiB [18:32] and 7z lets you not uncompress it [18:32] s/7z/bzip2 [18:33] found a for 7z to do a 7zcat and 7zless [18:33] *a script [18:34] exec 7z e -so -bd "$@" 2>/dev/null | cat [18:34] hm, doesn't look very efficient: piping 350 GiB? [18:34] anyway, I don't really know what I'm talking about, just repeating things I heard real experts say .-p [18:35] you might want to read https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l where this was discussed multiple times [19:06] I use wikitaxi for offline wikipedia which is a nice ready-made app but there's like one guy writing it and he doesn't update enough [19:15] godane: becauze xz is a fucking dickbitch to compress with [19:15] since it takes a billion more resources [19:50] Just slammed another 5gb of shareware CDs in [19:50] I'm trying to empty batcave out in the next week [19:50] Except Berlios and the current projects, while we determine ingestion [19:52] I notice the cds that have gone in in the past while don't have the auto-generated file listings and galleries [19:54] I have to initiate them. [19:54] Shortly, I'll write something to extract them. [19:54] Right now, I need the iso location of them to do it. [19:56] Like, what I SHOULD do is you give it an item name, it checks if the work has been done, and then it does it if there's nothing. [19:56] Then I can just glorb the rss feed [19:59] isn't there already a derive infrastructure for this sort of thing [20:00] Yeah, but adding derive jobs to the system is a pain in the ass [20:31] godane, Nemo_bis: wiki dumps compress even better when you first diff-compress them and then bzip the diffs. [20:31] godane: Nemo_bis: https://github.com/chronomex/wikiscraper is a tool that takes a wiki dump and turns it into a version-control repo [20:32] chronomex, are you interested in doing the opposite? :) [20:33] hmmm, maybe, what do you have in mind? [20:34] old diffs of Wikipedia in UseModWiki [20:34] ? do link [20:35] http://reagle.org/joseph/blog/social/wikipedia/10k-redux.html [20:35] Hm. Splinder tracker seems to be getting less reliable. Just had to restart about half a dozen out of 16 processes for failure to tell teh tracker the user was finished. [20:36] hmmm. [20:37] I'd think it was at my end, but my rsync up to batcave was uninterrupted and others reported the same problem before. [20:37] So that leaves the tracker looking like the weak link. [20:38] I'm away from my box right now but it's not showing up on the dashboard. [20:38] hm. [20:38] yes, and sometimes it also fails to give a user and dld-streamer stops adding processes [20:38] i don't think there's a huge problem with failing to notify the tracker occasionally. it'll cause duplication but it's better than saying you got something that you didn't actually get [20:39] True enough. But given that we're likely to be pressed for time, I'd like to avoid any duplication if possible. [20:39] dld-streamer is designed to abort gracefully when the tracker goes away. [20:39] yeah. I'll look into it when I get home in ~8 hours.. [20:39] Especially if the solution might be as simple as, "if the response times out, wait 5 seconds and try again" [20:40] I'd have to move a few things around, it's a 15-minute fix. [20:40] including testing. [20:40] It always gives me a new user right away when I restart so it's definitely transient. [20:40] yeah. it's transient. [20:40] yes, but it means that you have to restart the streamer a lot of times; I had to disable the automatic stop or I'd have had to restart it 14 times in the last ~2000 users [20:40] which is why I wasn't able to test it reliably [20:40] huh, lame. [20:41] Didn't we have the same problem with GV? Whichever fake-database-thing being used doesn't seem to be able to handle the mass of updates, or something like that. [20:41] well. streamer is much better than a bunch of clients in screen, that's for damn sure. [20:42] Maybe if you have 50 to deal with. My VM and ancientware machine can handle so little it's easy to pop between screen windows :) [20:42] heh [20:42] yeah [20:43] when I run 50, it spends a lot of cpu time in extract-urls.py [20:43] like, a lot. [20:43] Also handy for catching the "my blog domain has dashes" users that send it into a hoke-and-retry cycle [20:43] it chokes much on that? [20:43] s/hoke/choke [20:44] Not too often, but it gets a "bad" error when the standard-weenie DNS decides the subdomain can't exist, so it deletes the user and starts over. [20:44] ah. [20:44] I've had 2, so I assume they're out there at about a rate of 1 per 4000 [20:44] sounds reasonable given the data available [20:45] I take it nobody is doing memac? [20:45] And again, that's with dld-client being called directly. dld-stramer may catch and fail more gracefully. [20:45] well, nobody other than me [20:45] I added them to http://archiveteam.org/index.php?title=Splinder#Notes [20:45] dld-streamer didn't tell anything about it [20:45] DoubleJ: dld-streamer is mostly just a re-arranged dld-client. [20:46] OK. Didn't know if it was a wrapper or a rewrite. [20:46] yeah. it manages a pool of dld-single.sh's [20:47] If you're in the mood to test, see what it does with it:ermejointutt [20:47] I have to put some more robustness into its error handling [20:47] The blog is at -dituttounpo-.splinder.com which Linux boxen seem to be incapable of resolving. [20:48] on bsd: $ host -- -dituttounpo-.splinder.com [20:48] host: convert UTF-8 textname to IDN encoding: prohibited character found [20:48] Which starts the download-choke-delete-retry cycle forever [20:48] Yep. Probably the same error on Linux. But it's out there and resolves correctly on other OSes, and prevents the user from being completed. [20:49] fuckres. [20:49] I think it may be an older version of the spec? ISTR that dashes used to be OK in subdomains, but I might be remembering wrong. [20:49] "something of everything" (I wonder what's the exact translation) [20:49] *.splinder.com is CNAME aliased to blog.splinder.com. Can we bypass the DNS entirely? [20:49] You know bach, you tell me :) [20:50] bash [20:50] heh, it's more a wget question. [20:50] and I don't know very much bash, I just know how to program and read manpages ;) [20:53] btw. most of the logfiles that dld-streamer keeps around temporarily are because I didn't trust myself to handle work units properly, and wanted a record until they were successfully despatched [20:54]