[00:01] *** zenguy_pc has joined #archiveteam-bs [00:05] *** zenguy_pc has quit IRC (Read error: Connection reset by peer) [00:16] *** primus104 has quit IRC (Leaving.) [00:22] *** zenguy_pc has joined #archiveteam-bs [00:25] *** rduser has quit IRC (Read error: Operation timed out) [00:32] *** rduser has joined #archiveteam-bs [00:41] *** BlueMaxim has joined #archiveteam-bs [00:49] *** superkuh_ is now known as superkuh [01:11] *** Fusl has quit IRC (Ping timeout: 240 seconds) [01:20] *** Aranje has quit IRC (Quit: Three sheets to the wind) [01:32] *** aaaaaaaaa has quit IRC (Leaving) [01:51] *** dashcloud has quit IRC (Read error: Connection reset by peer) [01:52] *** dashcloud has joined #archiveteam-bs [04:21] *** JesseW has joined #archiveteam-bs [04:28] *** chfoo0 has joined #archiveteam-bs [04:30] *** chfoo has quit IRC (Ping timeout: 306 seconds) [04:31] *** zenguy_pc has quit IRC (Read error: Connection reset by peer) [04:41] *** chfoo0 is now known as chfoo [04:55] *** Fletcher has quit IRC (Ping timeout: 252 seconds) [04:56] *** diacope has quit IRC (Ping timeout: 252 seconds) [05:07] *** zenguy_pc has joined #archiveteam-bs [05:08] *** zenguy_pc has quit IRC (Read error: Connection reset by peer) [05:14] *** diacope has joined #archiveteam-bs [05:25] *** zenguy_pc has joined #archiveteam-bs [05:27] *** Fletcher has joined #archiveteam-bs [06:09] *** Fusl has joined #archiveteam-bs [06:29] *** Fusl has quit IRC (Contact: http://hallowe.lt/) [06:30] *** JesseW has quit IRC (Read error: Operation timed out) [06:30] *** Fusl has joined #archiveteam-bs [06:31] *** phuzion has quit IRC (Read error: Operation timed out) [06:40] *** phuzion has joined #archiveteam-bs [07:16] *** JesseW has joined #archiveteam-bs [07:31] *** schbirid has joined #archiveteam-bs [08:13] *** primus104 has joined #archiveteam-bs [08:14] *** JesseW has quit IRC (Read error: Operation timed out) [08:45] *** primus104 has quit IRC (Leaving.) [08:53] *** szalwia has quit IRC (Remote host closed the connection) [08:53] *** JSharp has quit IRC (Remote host closed the connection) [08:53] *** zyphlar has quit IRC (Remote host closed the connection) [08:53] *** Boltsie has quit IRC (Remote host closed the connection) [08:56] *** arkiver2 has joined #archiveteam-bs [09:01] *** JSharp has joined #archiveteam-bs [09:05] *** arkiver2 has quit IRC (Ping timeout: 252 seconds) [09:08] *** arkiver2 has joined #archiveteam-bs [09:14] *** zyphlar has joined #archiveteam-bs [09:48] *** Boltsie has joined #archiveteam-bs [09:50] *** szalwia has joined #archiveteam-bs [10:29] *** primus104 has joined #archiveteam-bs [10:33] *** RedType has quit IRC (Ping timeout: 483 seconds) [10:53] *** arkiver2 has quit IRC (Ping timeout: 252 seconds) [10:59] *** primus104 has quit IRC (Leaving.) [11:00] *** RedType has joined #archiveteam-bs [11:01] *** arkiver2 has joined #archiveteam-bs [11:29] i'm starting to download the JIBS News General broadcasts again [11:56] *** PurpleSym has joined #archiveteam-bs [12:17] *** primus104 has joined #archiveteam-bs [12:28] *** zenguy_pc has quit IRC (Read error: Connection reset by peer) [12:45] *** zenguy_pc has joined #archiveteam-bs [12:49] *** arkiver2 has quit IRC (Ping timeout: 252 seconds) [12:49] *** no2pencil has joined #archiveteam-bs [13:43] *** primus104 has quit IRC (Leaving.) [13:52] *** RichardG has quit IRC (Ping timeout: 252 seconds) [14:42] *** BlueMaxim has quit IRC (Quit: Leaving) [15:49] *** primus104 has joined #archiveteam-bs [15:50] *** JesseW has joined #archiveteam-bs [15:57] *** lytv has quit IRC (Ping timeout: 240 seconds) [15:57] *** lytv has joined #archiveteam-bs [16:19] *** JesseW has quit IRC (Read error: Operation timed out) [16:27] *** primus104 has quit IRC (Leaving.) [16:32] for the logs, an awful but working way to extract pdf downloads from http://www.researchgate.net/publication/ID urls: grep -Poh -m 1 'http.*pdf' 267864* | grep -v "/pdfjs/" | grep -v "{" | grep -v '"' [16:38] schbirid: sample link? [16:40] curl -s http://www.researchgate.net/publication/267864297_Ambient_assisted_living_with_dynamic_interaction_ensembles | grep -Poh -m 1 'http.*pdf' | grep -v "/pdfjs/" | grep -v "{" | grep -v '"' [16:41] if you can figure out how to make curl follow the redirect you can strip the text after the id [16:41] yes, i am tempted to start something... [16:41] lol [16:41] -L [16:42] ura nexXxt l3v3l haxx0r! [16:42] thanks [16:42] | xargs wget [16:42] ! [16:42] nah, wget profiles, archive profiles, grep profiles, wget pdfs, archive pdfs [16:42] * joepie91 bookmarks it with xargs wget anyway [16:43] /bookmark [16:43] that would be nice [16:43] schbirid: https://github.com/pindexis/marker [16:43] :p [16:43] there you go [16:43] terminal bookmarks with fuzzy search [16:43] (it's basically magic) [16:43] omg [16:43] schbirid: watch the video [16:43] :D [16:45] schbirid: even lets you jump from argument to argument [16:45] via % placeholders [16:45] author looks like a muslim though! [16:45] so you get something akin to shell functions, but in bookmark form [16:46] lol [16:46] gonna try it later [16:46] awesome! [16:46] schbirid: must be a bomb [16:46] :p [16:46] he might own a clock [16:46] you never know with THOSE PEOPLE [16:46] schbirid: caveat: must have zsh or bash 4.3+ [16:46] had to install bash 4.3 via Nix myself [16:46] cavewat [16:46] because openSUSE ships with 4.2 [16:46] *** lbft has quit IRC (Read error: Operation timed out) [16:46] bash --version [16:46] GNU bash, version 4.3.42(1)-release (x86_64-unknown-linux-gnu) [16:46] have to say, was a slightly unreal experience to get an error saying "your bash is too old" [16:46] my.. what? [16:47] you mean that thing actually still changes? [16:47] lol [16:47] :D [16:58] *** lbft has joined #archiveteam-bs [17:10] joepie91: at least you got an error [17:11] I didn't know that was possible, most times I just get obtuse syntax errors [17:16] *** primus104 has joined #archiveteam-bs [17:36] *** Lord_Nigh has quit IRC (ZNC - http://znc.in) [17:40] *** RichardG has joined #archiveteam-bs [17:57] *** ete has joined #archiveteam-bs [18:00] yipdw: it was the marker installer [18:00] :p [18:00] giving that errro [18:01] error * [18:02] ahh [18:29] yipdw: somebody telling you that bash has reasonable error reporting is about as plausible as somebody telling you they just shook hands with a dragon [18:42] *** Lord_Nigh has joined #archiveteam-bs [19:00] Debian Jessie is rubbish [19:00] HCross: how's that? [19:01] I was going to start on the Blingee grab, but I am sorting dependancy hell [19:01] HCross: what kind of dependencies? [19:02] the normal libgnutls-dev thing [19:02] HCross: sec [19:02] HCross: https://github.com/ArchiveTeam/standalone-readme-template#for-debianubuntu [19:02] hasn't made it into the blingee repo yet, apparently [19:02] I fixed that a week or two ago :P [19:03] ah, ive just been fighting with /etc/apt/sources.list [19:03] don't :P [19:03] *** limebyte has joined #archiveteam-bs [19:03] just use that 28 package [19:03] it'll fix it [19:03] * joepie91 waves at limebyte [19:03] * limebyte waves mornings [19:03] limebyte: so here's where long discussions, off-topic, etc go [19:03] my cat has big tits [19:03] like that [19:03] :P [19:04] HCross: took me way too long to figure out the fix for gnutls, btw [19:04] poor documentation++ [19:04] I spent the first 30 minutes under the impression that they simply stopped shipping libgnutls... [19:04] because apparently the name change isn't documented anywhere except for an obscure mailing list thread [19:05] limebyte: anyhow, is it running now? [19:05] still booting in screen [19:06] hope it goes away soon in background [19:06] limebyte: make a screenshot/paste? [19:06] also, you have to manually detach - ctrl+A D [19:06] acker rate limiting is active. We don't want to overload the site we're archiving, so we've limited the number of downloads per minute. Retrying after 300 seconds... [19:06] etc pp [19:06] ahhh [19:06] yeah, that's fine [19:07] just let it run [19:07] it'll work [19:07] a detached [19:07] good [19:07] the WI is nice [19:07] running it with ssh tunnel [19:07] limebyte: we do that on the task tracker when 1) we need to pause the project, or 2) the target site bans above a certain rate, or 3) the target site can't deal with the load [19:07] ? [19:07] oh, the web interface? [19:07] ya [19:07] limebyte, this is what I said about the we-dont-want-to-ddos-them [19:07] ah cmon [19:08] HCross: generally best to leave that determination to the tracker though :P [19:08] isnt a DDOS [19:08] HCross: so that it can be ramped up whenever necessary without asking people to reconfigure their shit [19:08] we call it a "distributed denial of preservation attack" [19:08] HCross: distributed preservation of service * [19:08] ah thats it [19:08] :) [19:09] im late to the party on this one then [19:10] limebyte: Please try to have 80 GB available on where you are running the grab scripts [19:11] Mount Total Used Avail Prcnt Graph [19:11] dang [19:11] arkiver: 80GB? surely we don't have items that big for blingee? [19:11] Not for blingee [19:11] But some google code project might be big [19:11] arkiver: oh, in general you mean [19:11] yes [19:11] Mount Total Used Avail Prcnt Graph [19:11] 1.79 TB 1.10 GB 1.79 TB 0.1% [----------] [19:11] enought storage so far [19:11] *** aaaaaaaaa has joined #archiveteam-bs [19:11] limebyte: looks good! [19:11] yeah got that baby yesterday [19:11] limebyte: please also make sure no websites or links are blocked/filtered [19:11] so not tor or any other proxy [19:12] nah [19:12] limebyte: datashack it was, right? [19:12] ya [19:12] same as HCross [19:12] yeah, should be all good then [19:12] * joepie91 picked up a dacentec box yesterday.. [19:12] wow [19:12] i could also use my Ikoula Node [19:12] one of those $20 ones :) [19:12] but already TOR/Yacy/Torrent running [19:13] quite high I/O load [19:13] 4TB for $20 \o/ [19:13] joepie91, I got one too and am setting it up now [19:13] hehe [19:13] joepie91, also 1TB for 5 [19:13] HCross: tried to install jessie. didn't realize that it would disable root login by default [19:13] but Intel Atom and just 100Mbit unmetered [19:13] http://status.x8e.net/mb.php <3 [19:13] need to add datashaky [19:13] HCross: and IP KVM confusingly shows 'no signal' when screen is asleep [19:13] HCross: so I was all "wtf? y it no work" [19:14] HCross: but that was apparently a bug, they hadn't added the "root login is disabled" warning to the Jessie template yet [19:14] lol [19:14] joepie91, I had to get support to install debian for me, their installer hung [19:14] HCross: oh, worked for me. but I went for single disk [19:14] 1x4TB [19:14] HCross, worked [19:14] used there install thingy [19:14] HCross: I believe the issue only occurs on multi-disk non-RAID setups [19:14] ah, they replied very early in the morning [19:14] got 2x2TB raid [19:14] worked [19:14] yeah, I got <10 minute responses, pretty okay with that [19:14] :P [19:14] like 5 min and running [19:15] limebyte: neat graphs [19:15] ya [19:15] and poof they went [19:15] lol [19:15] lol [19:15] limebyte: privacy badger is nuking the highcharts JS.. [19:15] should probably host that locally [19:15] okay gimme a sec [19:16] limebyte: privacy badger uses heuristics - ie. highcharts sends cookies and doesn't respect do-not-track header [19:16] :P [19:16] i know [19:16] actually try to defend [19:16] but at work also using that crap to track people [19:16] SEO and stuff [19:16] :( [19:17] (SEO is bullshit anyway) [19:17] ya [19:17] no2pencil: perhaps more practical to answer in here :P [19:17] no2pencil: are you running a Warrior VM, or manually? [19:18] It's been a while since I was active with a project, but the previous one I helped with used a VM iso [19:18] no2pencil: alright - you can still use the same VM/ISO (I assume you're running on a desktop/laptop) [19:18] no2pencil: and ideally pick "archiveteam's choice" - then it should automatically switch to whatever project has most priority at the moment [19:19] joepie91, any idea on best concurrency for a Kiderchire? [19:19] HCross: what CPU? [19:19] because those celerons max out a core when you so much as look at them funny [19:20] :P [19:20] i hope its better now [19:20] all local [19:20] joepie91, it isnt a celeron.. its much worse http://www.online.net/en/dedicated-server/dedibox-scg2 [19:20] cpu on kidichire is crap [19:20] that's why it's best to leave the case on the PC, so the celeron can't see you looking at it funny [19:20] HCross: er, sorry, that's what I meant. yeah, I wouldn't run more than maybe 2-4 threads on that [19:20] it is really, REALLY bad [19:21] HCross: in a daze of insanity I once tried running one of those as an rsync target [19:21] it was not a success [19:21] lol [19:21] Yep, I used it as my VPN server earlier and it reduced a 152Mbps connection at 15Mbps [19:21] I think it got to some 800+ load average before I could finally get a `killall` in [19:23] joepie91, the scaleways do these archive jobs very very well [19:24] HCross: how much do those cost again [19:24] €3 plus Vat I think [19:25] for? [19:25] (specs) [19:26] Some quad core ARM A7, 2GB RAM and 50GB ssd [19:26] and its €1 per 50GB ssd [19:26] reasonable [19:27] 2,99 + VAT [19:29] where's that from, HCross? [19:29] https://www.scaleway.com/ [19:29] ah [19:30] limebyte, watch http://tracker.archiveteam.org/blingee/ [19:30] lame [19:30] dont see my name [19:30] I jsut saw it [19:30] A THERE [19:30] I AM FAMOUS [19:30] YAY [19:31] you should have seen the start of the skilfeed grab as I got in about 5 mins before the others [19:31] I'd get a Scaleway if it didn't require a CC... [19:31] it does [19:31] sadly [19:31] Ive been thinking of reselling them and accepting PP [19:31] but they also do SEPA now [19:31] limebyte: doesn't list it on the site? [19:32] limebyte, online do but scaleway doesnt I think [19:32] (also, why can't they just take paypal and bitcoin like everybody else) [19:32] or was SEPA for online [19:32] ah online [19:32] any list of still running projects? [19:35] ive got 17 wgetdownload in progress [19:36] okay dat wget [19:36] pulls cpu as fuck [19:36] Load 2.4 [19:36] neat [19:37] if needs be, crank the concurrency down [19:37] nah [19:37] good good [19:37] My Dacentec atm http://harrycross.me/4c8.png [19:40] limebyte: if you look at tracker.archiveteam.org there is a list a the bottom of the page. The one marked with a star is the AT choice and considered the top priority [19:41] okay good [19:41] thx [19:41] also, if nothing else seems to be running, URLteam almost always is [19:41] * limebyte hugs aaaaaaaaa [19:42] aaaaaaaaa: when the tracker isn't down ;) [19:42] I said "almost" [19:42] aaaaaaaaa: though the whole "oh, it's the other week, the tracker is down again" seems to have ceased since it moved to the central tracker [19:42] and it's now generally actually available [19:42] lol [19:42] the tracker is more reliable than Skype atm [19:42] HCross: not a very tall order... [19:43] ... [19:43] ... [19:43] aaaaaaaaa: THERE'S A STAR ON THE TRACKER INDEX [19:43] I have been living a lie [19:43] how did I not know about this.. [19:45] aaaaaaaaa: who operates the urlteam tracker? [19:45] I found a broken :P [19:45] xmc owns the box chfoo runs the tracker [19:47] hm. not sure whether it's a box or tracker issue [19:47] xmc: chfoo: I found a broken; at http://tracker.archiveteam.org:1337/status, at the bottom, the git hash says "Command '['git', 'rev-parse', 'HEAD']' returned non-zero exit status 128" [19:51] oktay [19:51] other projects dont seem to have debian install [19:51] just replace urls and i am fine i guess? [19:52] limebyte: which projects? [19:52] afaik blingee and urlteam are the only two currently active warrior projects [19:53] also skillfeeed [19:53] https://github.com/ArchiveTeam/terroroftinytown [19:53] https://github.com/ArchiveTeam/terroroftinytown-client-grab [19:54] the other is the tracker, not the client code [19:55] kay [19:55] https://github.com/ArchiveTeam/skillfeed-grab but is also active right? [19:55] limebyte: Skillfeed is finished [19:55] limebyte: see http://tracker.archiveteam.org/skillfeed/ [19:55] 0 to do [19:55] dang [19:55] limebyte: also, terroroftinytown is the urlteam one :P [19:56] i know joepie91 [19:56] and don't worry, there'll be more projects later :D [19:57] these few weeks are going to be busy [19:59] http://status.x8e.net/mb.php datashaky added [19:59] neat [19:59] 0.5MB/s its trying [20:01] dang bug [20:04] wow that was quick [20:04] need to reboot do kernel updatez [20:06] *** schbirid has quit IRC (Quit: Leaving) [20:09] HCross: why is that? [20:10] I thought we had a load of stuff to do, with GCode and comcast [20:11] HCross: comcast? [20:11] (and yeah, gcode will be big) [20:13] joepie91, Comcast web hosting is closing soon [20:13] joepie91, http://archiveteam.org/index.php?title=Comcast_Personal_Web_Pages [20:16] so Swift's block shorthand and $-convention for closure arguments end up being super-useful and I hope something like them makes it into Rust or C++ [20:16] [20:17] yipdw: so, like JS? :P [20:17] sort of [20:17] I am burnt out on Ruby and Javascript though [20:17] yipdw: it was half a joke :) [20:17] operations ended up doing that [20:21] finally... [20:22] joepie91: I was able to get the headless version running on a vps, will have to get the VM iso redownloaded & installed when I get home [20:23] no2pencil: oh. for servers, the manual method is usually a better fit... [20:23] no2pencil: or was that what you meant? [20:25] yeah, I have a couple of vps'es [20:26] everytime I come back to help with some projects, you guys have gone leaps & bounds with these setups, it's great. [20:42] *** edsu_ has quit IRC (Read error: Operation timed out) [20:45] limebyte, good work. You are appearing regularly in the list [20:48] wow [20:48] ya HCross [20:50] what whas the uri for UrlTeam again? [20:51] *** edsu has joined #archiveteam-bs [20:53] https://github.com/ArchiveTeam/terroroftinytown-client-grab [20:53] tracker uri [20:54] http://tracker.archiveteam.org:1337 [20:54] thx [20:56] *** PurpleSym has quit IRC (Remote host closed the connection) [21:30] JIBS News General for 2011-01 is getting uploaded [21:31] just know that 2011-01-20 episode doesn't exist anymore [23:53] *** c_b has joined #archiveteam-bs [23:55] I like how we get the same people with the same problems showing up in #archiveteam [23:56] it's like NPCs in an RPG