Time |
Nickname |
Message |
00:00
🔗
|
yipdw |
more seriously, I think we could actually use git-annex at my current workplace for versioning slide scans |
00:00
🔗
|
yipdw |
or perhaps annotation data on that |
00:00
🔗
|
yipdw |
slide scans are a few hundred megabytes each |
00:01
🔗
|
chronomex |
that's kind of big |
00:01
🔗
|
chronomex |
what sort of instrument do you use to scan? |
00:01
🔗
|
yipdw |
they're massive, massive images |
00:02
🔗
|
chronomex |
and what sort of originals? |
00:02
🔗
|
yipdw |
I think |
00:02
🔗
|
yipdw |
we're using a NanoZoomer 2.0 slide scanner here |
00:03
🔗
|
yipdw |
on average, it's not quite as bad as a few hundred megabytes |
00:03
🔗
|
yipdw |
but when you combine multiple focal planes with the highest image quality, yeah, it can get there |
00:03
🔗
|
chronomex |
aha. |
00:03
🔗
|
yipdw |
the idea is that you should capture enough data from the slide to permit diagnoses to be made from the capture |
00:04
🔗
|
yipdw |
and the bar for that is "what a histologist can see through a microscope" |
00:04
🔗
|
yipdw |
which is quite high :P |
00:04
🔗
|
chronomex |
a friend and I have cobbled together a high-speed slide scanner, using a carousel slide projector + low wattage bulb + ground glass screen behind the slide, dslr where the projector lens normally lives |
00:04
🔗
|
chronomex |
ahaha different slides :P |
00:04
🔗
|
yipdw |
heh yeah |
00:05
🔗
|
yipdw |
I forgot "slide" had another meaning |
00:05
🔗
|
chronomex |
we've gotten really good results with a not-very-fancy camera |
00:05
🔗
|
yipdw |
ah, here we go |
00:05
🔗
|
yipdw |
http://sales.hamamatsu.com/en/products/system-division/virtual-microscopy/index.php?id=13222680 |
00:05
🔗
|
yipdw |
maximum resolution is 0.23 micrometers/piel |
00:05
🔗
|
yipdw |
pixel |
00:05
🔗
|
chronomex |
yow |
00:05
🔗
|
yipdw |
so for a 26x76mm slide, yeah |
00:05
🔗
|
yipdw |
work that out |
00:06
🔗
|
yipdw |
generally the person scanning the slide will select a region of interest so that they don't have to wait forfuckingever to get an image |
01:42
🔗
|
arrith |
http://torrentfreak.com/book-publishers-shut-down-library-nu-and-ifile-it-120215/ |
01:42
🔗
|
SketchCow |
According to the complaint, the sites offered users access to 400,000 e-books and made more than $11 million in revenue in the process. |
01:42
🔗
|
SketchCow |
See. |
01:42
🔗
|
SketchCow |
This is the thing. |
01:43
🔗
|
SketchCow |
It is so hard for me to go "OH NO A DIGITAL LIBRARY OF ALEXANDRIA IS GONE" |
01:43
🔗
|
SketchCow |
I work at one, thank you very much |
01:43
🔗
|
arrith |
i'm just wondering if that stuff is really gone :( |
01:43
🔗
|
arrith |
links that is |
01:44
🔗
|
arrith |
if library.nu at least comes back in some form one could crawl it, but this was crazy-sudden |
01:44
🔗
|
chronomex |
but THINK OF THE CHILDREN |
01:44
🔗
|
SketchCow |
Oh I am |
01:44
🔗
|
SketchCow |
OK, finished |
01:45
🔗
|
* |
SketchCow zips up |
01:45
🔗
|
SketchCow |
So, how long has git-annex been around? |
01:45
🔗
|
SketchCow |
I had someone drop this on me, so it's all new, but obviously it is rather mature. |
01:45
🔗
|
chronomex |
I think two years? |
01:45
🔗
|
chronomex |
maybe three |
01:47
🔗
|
arrith |
well their gitweb only goes back to 2010-10-09 |
01:48
🔗
|
arrith |
which is around the time articles about it started popping up |
01:48
🔗
|
kennethre |
arrith: well that's 1/3rd the lifetime of git |
01:48
🔗
|
kennethre |
so *a long time* suffices ;) |
01:48
🔗
|
arrith |
git-annex is not some flaky script that was quickly thrown together. I wrote it in Haskell because I wanted it to be solid and to compile down to a binary. And it has a fairly extensive test suite. (Don't be fooled by "make test" only showing a few dozen test cases; each test involves checking dozens to hundreds of assertions.) |
01:48
🔗
|
arrith |
from http://git-annex.branchable.com/not/ |
01:48
🔗
|
arrith |
the dev seems to be pretty capable is why i pasted that |
01:49
🔗
|
chronomex |
that's about the time I first heard about it; I lurk on the vcs-home mailing list which is where it was announced iirc |
01:50
🔗
|
chronomex |
arrith: I believe closure is joey hess, the author |
01:50
🔗
|
arrith |
oh wow, that's neat |
01:51
🔗
|
arrith |
20 Oct 2010 is the earliest date in the debian changelog here: http://packages.debian.org/changelogs/pool/main/g/git-annex/git-annex_3.20120123/changelog |
01:51
🔗
|
arrith |
so yeah i'm going with late 2010 |
01:51
🔗
|
chronomex |
seems reasonable |
01:53
🔗
|
closure |
jesus 1/3 the lifetime of git?! |
01:55
🔗
|
SketchCow |
What the fuck kind of time measurement is that. |
01:55
🔗
|
SketchCow |
We'll be done with the project in .4 git-annex lifetimes |
02:06
🔗
|
DFJustin |
all of library.nu's actual content was hosted on other filehost sites |
02:07
🔗
|
chronomex |
like megaupload |
02:07
🔗
|
chronomex |
we're fucked |
02:08
🔗
|
arrith |
DFJustin: yeah, just need the links |
02:08
🔗
|
arrith |
almost like tpb and magnet links |
02:08
🔗
|
DFJustin |
but the alexandria comparisons are pretty silly because all of the stuff still exists in print in regular libraries |
02:09
🔗
|
arrith |
although there is some information in torrent files that won't be available through magnet links alone, tracker urls for example |
02:16
🔗
|
closure |
http://pastebin.com/NhA3VPhK ... I'm just saying |
02:34
🔗
|
SketchCow |
https://plus.google.com/hangouts/extras/talk.google.com/jason's%2520incredibly%2520boring%2520clubhouse?authuser=0&hl=en&eid= |
02:56
🔗
|
Archivis2 |
Blargh, I hate it when I have to repair my main desktop :( |
02:56
🔗
|
Archivis2 |
At least this channel is publicly logged |
02:59
🔗
|
Archivis2 |
closure: Been playing/learning about git-annex all evening |
02:59
🔗
|
Archivis2 |
This is excellent! |
02:59
🔗
|
Archivis2 |
Currently making a repo containing all archiveteam uploads on archive.org |
03:00
🔗
|
Archivis2 |
(pointing to web remotes) |
03:00
🔗
|
Archivis2 |
Mostly just to practice using it, but who knows, might be useful |
03:01
🔗
|
Archivis2 |
closure: wrt "file sharing", isn't it just adding other people's remotes and vice versa? |
03:01
🔗
|
Archivis2 |
(of course, they need to be trusted people since they need something like ssh access over git-annex-shell) |
03:01
🔗
|
closure |
well yeah, basically |
03:02
🔗
|
Archivis2 |
s/trusted/semi-trusted/ |
03:02
🔗
|
closure |
you can also put git repos on http:// and no logins needed |
03:02
🔗
|
closure |
or some other things |
03:02
🔗
|
Archivis2 |
yeah |
03:02
🔗
|
Archivis2 |
But if you put repos on http, how would you also distribute the files? |
03:02
🔗
|
closure |
by http |
03:02
🔗
|
closure |
in the same directory |
03:03
🔗
|
Archivis2 |
(Since they're not actually wrapped in the git repo) |
03:03
🔗
|
Archivis2 |
Oh, okay, just curious |
03:03
🔗
|
closure |
they're in .git/annex/objects/ which is accessible via http if you put .git up for http |
03:03
🔗
|
Archivis2 |
I'm only familiar with using git daemon to clone |
03:04
🔗
|
Archivis2 |
So if I git clone a repo, I end up with all the files that are "in" that copy of it at that time? |
03:04
🔗
|
closure |
no, it ends up empty, you have to get the files you want in that clone |
03:05
🔗
|
Archivis2 |
So, how would you do that over http? |
03:05
🔗
|
Archivis2 |
Just make .got/annex/objects web accessible, and mirror it? |
03:05
🔗
|
Archivis2 |
s/got/git/ |
03:06
🔗
|
closure |
you say "git annex get foo" and it goes and gets it, if you cloned from http:// it knows where to go |
03:06
🔗
|
Archivis2 |
Oh, wow |
03:06
🔗
|
Archivis2 |
That's rad! |
03:06
🔗
|
Archivis2 |
Same with a clone over git://, or only http? |
03:07
🔗
|
closure |
same with any clone, *except* for git:// actually |
03:07
🔗
|
closure |
(because git:// protocol can't transfer arbitrary files) |
03:07
🔗
|
closure |
but over ssh, sure |
03:07
🔗
|
closure |
or rsync |
03:07
🔗
|
Archivis2 |
Oh, okay. |
03:08
🔗
|
Archivis2 |
What's the easiest way to expose a git repo over http? |
03:08
🔗
|
Archivis2 |
(if you have an opinion) |
03:08
🔗
|
closure |
well, I think you want to make a separate, bare repo, and there's this hook you have to enable. bit of a bother really |
03:09
🔗
|
Archivis2 |
oh, I see |
03:09
🔗
|
closure |
course for your repo of all the archiveteam stuff, you could just put it on github |
03:09
🔗
|
Archivis2 |
yeah, 's what I planned to do |
03:09
🔗
|
closure |
since you're telling it where to get all the files from the web |
03:10
🔗
|
Archivis2 |
Since everything's web-remote |
03:10
🔗
|
Archivis2 |
Yep :) |
03:10
🔗
|
Archivis2 |
This is incredibly awesome, btw |
03:10
🔗
|
chronomex |
that's sweet. |
03:10
🔗
|
chronomex |
hmmm. |
03:10
🔗
|
Archivis2 |
Athough now I'm wrestling cabal |
03:10
🔗
|
chronomex |
looks like git-annex will solve some stupid problems I have |
03:10
🔗
|
Archivis2 |
(trying to compile the latest git-annex, since the hackage version doesn't have the --file flag for addurl) |
03:11
🔗
|
closure |
I'm not very fond of cabal. |
03:11
🔗
|
closure |
with what version of ghc are you building it? |
03:11
🔗
|
Archivis2 |
Could not deduce (Show a) arising from a use of `showHex' |
03:11
🔗
|
Archivis2 |
Data/Digest/SHA2.hs:111:4: |
03:11
🔗
|
Archivis2 |
[ 1 of 26] Compiling Data.Digest.SHA2 ( Data/Digest/SHA2.hs, dist/build/Data/Digest/SHA2.o ) |
03:11
🔗
|
Archivis2 |
bound by the instance declaration at Data/Digest/SHA2.hs:109:10-39 |
03:11
🔗
|
Archivis2 |
from the context (Integral a) |
03:11
🔗
|
Archivis2 |
7.4.1 |
03:11
🔗
|
Archivis2 |
Latest |
03:11
🔗
|
Archivis2 |
(the problem isn't in your thing, it's in one of the deps) |
03:11
🔗
|
closure |
yeah, I had the same failre recently. Something broken there |
03:12
🔗
|
Archivis2 |
did you work around it? |
03:12
🔗
|
closure |
I can't remember |
03:12
🔗
|
Archivis2 |
:P |
03:12
🔗
|
Archivis2 |
I suppose I could build without hs3 |
03:13
🔗
|
closure |
yep |
03:13
🔗
|
closure |
in fact, I think that's what I did.. git merge no-s3 |
03:15
🔗
|
Archivis2 |
I feel like I'm forgetting something dumb |
03:15
🔗
|
Archivis2 |
V |
03:15
🔗
|
Archivis2 |
0 3:15AM:abuie@abuie-dev:~/master 23266 Ã git merge no-s3 |
03:15
🔗
|
Archivis2 |
fatal: 'no-s3' does not point to a commit |
03:16
🔗
|
closure |
wtf, you're underscor? |
03:16
🔗
|
Archivis2 |
oh, yeah |
03:16
🔗
|
Archivis2 |
sorry |
03:16
🔗
|
closure |
try origin/no-s3 |
03:17
🔗
|
underscor |
Wheeeee |
03:17
🔗
|
underscor |
Thanks |
03:17
🔗
|
closure |
now SketchCow needs to come in here as overfiend or something, and our collection of name confusion would be complete |
03:18
🔗
|
underscor |
haha |
03:19
🔗
|
underscor |
oops, forgot pcre-light |
03:19
🔗
|
underscor |
With addurl --fast, does WORM info get recorded? |
03:20
🔗
|
underscor |
(I know you can't do a sha backend with fast, but I wasn't sure about worm) |
03:21
🔗
|
closure |
with the new version and --fast, it records the file size. Which is basically all WORM does |
03:21
🔗
|
closure |
it doesn't use that backend, but it's the same level of assurance (ie, not much) |
03:22
🔗
|
closure |
note that you can always `git annex migrate` later and it will pull it down from the web and convert to a checksum |
03:23
🔗
|
underscor |
Oh, okay, excellent |
03:23
🔗
|
underscor |
jfeifjsepfjdpsfjdkfjweklfjsdcx |
03:23
🔗
|
underscor |
git-annex: unrecognized option `--file=boingboing-2000-2005_files.xml' |
03:24
🔗
|
closure |
git annex version? |
03:25
🔗
|
underscor |
git-annex version: 3.20120124 |
03:25
🔗
|
underscor |
whoops |
03:25
🔗
|
underscor |
My ruby $PATH and shell $PATH didn't match up |
03:25
🔗
|
underscor |
Working now |
03:26
🔗
|
underscor |
Heh, sorry |
03:38
🔗
|
dcmorton |
is batcave available for rsync'ing mobileme users again? drive is getting dangerously full |
03:38
🔗
|
closure |
someone drove the batmobile thru the wall of batcave, I hear there's a new bat location somewhere |
03:40
🔗
|
underscor |
dcmorton: git pull |
03:40
🔗
|
underscor |
then you can run the uploader |
03:44
🔗
|
dcmorton |
underscor: got it.. thanks |
03:44
🔗
|
underscor |
np |
03:49
🔗
|
underscor |
closure: I'm imagining a giant git-annex repo with everything in archive.org |
03:49
🔗
|
underscor |
:D |
03:49
🔗
|
underscor |
That would be really neat |
03:50
🔗
|
underscor |
I wonder how well it would scale to that though |
03:50
🔗
|
underscor |
heh |
03:52
🔗
|
closure |
you hit git scalability issues eventually |
03:52
🔗
|
closure |
I've been working last 3 days in scaling git |
03:52
🔗
|
closure |
git-annex to millions of files.. it does. but git, not so much :) |
03:53
🔗
|
* |
closure has a directory with 300 copies of the linux kernel source tree in it. takes a while to rm |
03:54
🔗
|
underscor |
haha |
03:54
🔗
|
underscor |
Where does the problem in git lie? |
03:54
🔗
|
closure |
yay for lvm, so I can just nuke the volume |
03:54
🔗
|
underscor |
Just inefficiency with an index of millions of files? |
03:54
🔗
|
closure |
oh, it keeps every file in .git/index and rewrites it all the time |
03:55
🔗
|
closure |
some other stuff. Facebook was complaining about this, it seems their source tree is insane and too big for git |
03:55
🔗
|
underscor |
haha |
03:55
🔗
|
underscor |
damn |
03:55
🔗
|
underscor |
hmm |
03:55
🔗
|
underscor |
Is there a way to make this work? |
03:55
🔗
|
underscor |
git annex addurl --fast --file=NUMBERS/geocities-3-d.7z.001 http://archive.org/download/2009-archiveteam-geocities-part1/NUMBERS/geocities-3-d.7z.001 |
03:56
🔗
|
closure |
what, to create the directory? |
03:56
🔗
|
underscor |
Right now it spits out an angry error about a nonexistent directory |
03:56
🔗
|
underscor |
Yeah |
03:56
🔗
|
underscor |
I mean, I can write logic to create the directories, and cd in to each one, etcetera |
03:56
🔗
|
underscor |
But that feels real clunky |
03:58
🔗
|
closure |
will fix |
03:58
🔗
|
underscor |
\o/ |
03:58
🔗
|
underscor |
Thanks! |
04:00
🔗
|
underscor |
Out of curiosity, why'd you choose haskell? |
04:00
🔗
|
underscor |
(I personally love the language, but I know there are others that don't agree) |
04:05
🔗
|
closure |
fixed. |
04:05
🔗
|
closure |
because it was time to learn haskell and also I wanted something solid |
04:06
🔗
|
closure |
I didn't know you were into haskell |
04:12
🔗
|
yipdw |
"a monoid is a monad in the category of endofunctors, what's the problem?" |
04:12
🔗
|
yipdw |
er |
04:12
🔗
|
yipdw |
monad -> monoid |
04:13
🔗
|
yipdw |
whoops |
04:13
🔗
|
underscor |
closure: Yeah, I got into it this summer when I was in summer residential governor's school |
04:14
🔗
|
underscor |
one of my classes was mathematical problem solving, and haskell's lazy evaluation and excellent iterable abilities let me solve problems 20 times faster than anyone else in the class |
04:14
🔗
|
underscor |
(who wasn't using haskell) |
04:14
🔗
|
* |
closure hits yipdw with a typoclassopedia |
04:14
🔗
|
underscor |
I haven't done anything beyond write math programs in it though |
04:15
🔗
|
yipdw |
underscor: heh, that's about my exposure too -- I've been using it for Project Euler |
04:15
🔗
|
yipdw |
for some weird reason |
04:15
🔗
|
closure |
well damn, I wish I'd known, I could have had you slinging git-annex code |
04:16
🔗
|
yipdw |
underscor: also, Paul Hudak's The Haskell School of Expression is most excellent |
04:16
🔗
|
yipdw |
I will forever love fibs = 1:1:zipWith (+) fibs (tail fibs) |
04:16
🔗
|
underscor |
Ooh, I'll have to check it out |
04:16
🔗
|
underscor |
Yeah, that was the first problem we had to solve |
04:16
🔗
|
underscor |
10 thousandth fib number |
04:17
🔗
|
underscor |
Only took me a few minutes, everyone else spent >45 minutes |
04:17
🔗
|
underscor |
closure: Hehee |
04:18
🔗
|
underscor |
I need to learn about how the rest of it works, though |
04:18
🔗
|
underscor |
Like type declarations and stuff |
04:18
🔗
|
closure |
hmm, I need to get ahold of the Haskell School of Expression |
04:18
🔗
|
underscor |
My exposure's pretty much limited to fuckery in ghci |
04:18
🔗
|
yipdw |
underscor: try executing fibs !! 10000 on that definition, it will fly |
04:19
🔗
|
underscor |
I know :D |
04:19
🔗
|
closure |
yeah, I've been starting to try to learn about dependant types and type level programming |
04:19
🔗
|
yipdw |
on my Xeon it completes in something I can't measure |
04:19
🔗
|
yipdw |
the reason why it works is also mind-boggling (to me anyway) |
04:19
🔗
|
yipdw |
lazy evaluation up the ass |
04:19
🔗
|
underscor |
fucking delicious :D |
04:19
🔗
|
Coderjoe |
closure: is there a way to push git-annex files to another repository? |
04:20
🔗
|
closure |
Coderjoe: git annex copy foo --to reponame |
04:20
🔗
|
underscor |
Coderjoe: Yeah, git annex copy file --to remotename |
04:20
🔗
|
closure |
after you set up a git remote for it |
04:20
🔗
|
underscor |
closure: Damn :P |
04:20
🔗
|
Coderjoe |
and what protocols does it handle? |
04:20
🔗
|
Coderjoe |
and is there an ability to copy multiple files? |
04:20
🔗
|
closure |
any protocol that can be used for a normal git remote (except git://) .. ssh, rsync, http |
04:21
🔗
|
closure |
yes, foo can be a file or a directory |
04:21
🔗
|
closure |
or any number of either |
04:21
🔗
|
closure |
or leave it off to do the whole current directory :) |
04:21
🔗
|
underscor |
You can push via http? |
04:21
🔗
|
underscor |
Damn, never new that |
04:21
🔗
|
underscor |
+k |
04:22
🔗
|
closure |
um, no, you can't upload bia http |
04:22
🔗
|
Coderjoe |
aw |
04:22
🔗
|
closure |
well, I don't support WEBDAV yet at least.. |
04:22
🔗
|
Coderjoe |
because http can do it... |
04:23
🔗
|
closure |
true, but server side it's a bit of a nightmare |
04:23
🔗
|
underscor |
yipdw: closure: No Haskell School of Expression, but any other haskell books here catch your eye? http://hastebin.com/tahewokuco.coffee |
04:23
🔗
|
underscor |
Sorry for the gross formatting |
04:24
🔗
|
yipdw |
underscor: I haven't actually read any of those, though I did meet Bryan O' Sullivan at Erlang Factory once |
04:24
🔗
|
yipdw |
he's a pretty swell guy |
04:24
🔗
|
yipdw |
:P |
04:24
🔗
|
yipdw |
so I guess his Real World Haskell book is probably good |
04:24
🔗
|
underscor |
That's neat |
04:24
🔗
|
closure |
I've been meaning to look at the Bird too |
04:24
🔗
|
closure |
my name is in RWH :) |
04:24
🔗
|
underscor |
I suppose I was coming more from do-you-want-a-copy-of-them |
04:24
🔗
|
yipdw |
ohh |
04:24
🔗
|
yipdw |
illicit |
04:25
🔗
|
yipdw |
I SEE |
04:25
🔗
|
yipdw |
hmm |
04:25
🔗
|
closure |
um, Haskell School of expression is listed there :P |
04:25
🔗
|
yipdw |
the thing I like about HSE is that it uses functional programming for applications that IMO one does not see very often |
04:25
🔗
|
yipdw |
it actually focuses on FRP |
04:25
🔗
|
underscor |
Woah, look how blind I am |
04:25
🔗
|
yipdw |
which is pretty need for an introductory book |
04:25
🔗
|
yipdw |
neat, too |
04:25
🔗
|
underscor |
yipdw: Yeah, mildly illicit :P |
04:26
🔗
|
underscor |
This is basically library.nu, except private |
04:26
🔗
|
yipdw |
closure: you're cited in there? |
04:26
🔗
|
closure |
reviewer |
04:26
🔗
|
yipdw |
ahh neat |
04:26
🔗
|
closure |
along with like 500 other people |
04:27
🔗
|
Archivis2 |
Whoops, client crashed |
04:28
🔗
|
underscor |
Oh, yeah, I remember seeing your name, closure |
04:28
🔗
|
underscor |
I was like "I know that guy!" |
04:28
🔗
|
underscor |
haha |
04:28
🔗
|
underscor |
(yes, I'm that freak who reads all the reviews) |
04:29
🔗
|
underscor |
http://research.microsoft.com/en-us/um/people/simonpj/papers/history-of-haskell/index.htm |
04:29
🔗
|
underscor |
Looks interesting |
04:30
🔗
|
closure |
yes, I enjoyed that one |
04:30
🔗
|
yipdw |
Simon Peyton-Jones sounds like a supervillain's name |
04:30
🔗
|
yipdw |
along with R. Kent Dybvig |
04:30
🔗
|
closure |
Found: |
04:31
🔗
|
closure |
Why not: |
04:31
🔗
|
closure |
fmap a (getStale file) |
04:31
🔗
|
closure |
getStale file >>= return . a |
04:31
🔗
|
closure |
man, I love hlint |
04:31
🔗
|
underscor |
jvdksvjlewjvldsjvkdjsvldsvlkds |
04:31
🔗
|
closure |
you should find some SPJ talks. They're intorductory, but he's one of the best presenters I've ever seen |
04:31
🔗
|
underscor |
It's unseeded right now |
04:31
🔗
|
yipdw |
oh, hlint suggests alternative constructions? |
04:31
🔗
|
underscor |
I'll link it when it finishes downloading |
04:31
🔗
|
underscor |
Probably overnight |
04:42
🔗
|
closure |
hmm, a <$> getStale file is better though. hlint must not like applicatives |
04:51
🔗
|
underscor |
closure: yipdw: http://ksnd.it/dl/6613c98564e |
04:51
🔗
|
underscor |
It's a djvu |
04:53
🔗
|
closure |
expired request |
04:53
🔗
|
underscor |
Weird |
04:53
🔗
|
underscor |
One sec |
04:53
🔗
|
DFJustin |
works here |
04:53
🔗
|
underscor |
http://ksnd.it/v/6613c98564e |
04:54
🔗
|
underscor |
^ closure |
04:54
🔗
|
DFJustin |
damn wasted too much time listening to jason, now I have to do things |
04:55
🔗
|
underscor |
hahahah |
04:55
🔗
|
underscor |
00000000-0000-0000-0000-000000000001 -- web |
04:55
🔗
|
underscor |
85019413-6049-441c-a4a9-2f17dc0e734a -- here (ArchiveTeam Releases (@ IA)) |
04:55
🔗
|
underscor |
semitrusted repositories: 2 |
04:55
🔗
|
underscor |
untrusted repositories: 0 |
04:55
🔗
|
underscor |
dead repositories: 0 |
04:55
🔗
|
underscor |
local annex keys: 0 |
04:55
🔗
|
underscor |
local annex size: 0 bytes |
04:55
🔗
|
underscor |
known annex keys: 276 |
04:55
🔗
|
underscor |
known annex size: 57 gigabytes |
04:55
🔗
|
underscor |
backend usage: |
04:55
🔗
|
underscor |
URL: 276 |
04:55
🔗
|
underscor |
closure: Working beautifully now with the directory fix |
04:55
🔗
|
underscor |
Thanks! |
04:56
🔗
|
closure |
personally, and especially for the archiveteam repo, I "git annex untrust web" |
04:56
🔗
|
closure |
although if it's all archive.org urls, you *may* trust it :P |
04:56
🔗
|
underscor |
hehe |
04:56
🔗
|
underscor |
I planned on doing that, but after I add everything |
04:57
🔗
|
underscor |
Not really for any particular reason, I suppose |
04:57
🔗
|
underscor |
Makes it so I don't have to force on drop though |
05:27
🔗
|
underscor |
Wow! |
05:27
🔗
|
underscor |
where: &w_collection=*archiveteam* | size: 30,266,111,202 KB| |
05:27
🔗
|
underscor |
That's incredible! |
05:28
🔗
|
underscor |
Of course, mobileme will nearly increase it 10fold |
05:28
🔗
|
underscor |
But still! |
05:33
🔗
|
DFJustin |
kekekek |
05:33
🔗
|
DFJustin |
what's the size of cdbbsarchive out of curiosity |
05:35
🔗
|
underscor |
gimme a sec |
05:35
🔗
|
underscor |
btw, this is where fos lives, if y'all are curious about stats |
05:35
🔗
|
underscor |
http://ia700108.us.archive.org:8088/mrtg/ |
05:35
🔗
|
underscor |
You can see where someone started uploading mobileme |
05:35
🔗
|
underscor |
probably dcmorton |
05:35
🔗
|
underscor |
haha |
05:36
🔗
|
underscor |
DFJustin: where: &w_collection=cdbbsarchive | size: 366,592,842 KB |
05:36
🔗
|
underscor |
That would be pretty cool to have as a git-annex repo too |
05:38
🔗
|
DFJustin |
hehe |
05:39
🔗
|
underscor |
(Recording state in git...) |
05:39
🔗
|
underscor |
1 5:39AM:abuie@abuie-dev:~/cdbbsarchive 23540 Ã ruby ../ia_annex.rb Gold_II |
05:39
🔗
|
underscor |
Gold_II is not a collection. It is, in fact, a software |
05:39
🔗
|
underscor |
Mirroring Gold_II because its parent is cdbbsarchive |
05:39
🔗
|
underscor |
addurl GOLD_II.cdr ok |
05:39
🔗
|
underscor |
addurl GOLD_II.jpg ok |
05:39
🔗
|
underscor |
Wheee |
05:39
🔗
|
DFJustin |
git clone ALLSHAREWARE |
05:39
🔗
|
underscor |
That's how it'll be once this finishes ;P |
05:40
🔗
|
DFJustin |
guess this fulfills jason's wish for an easy way to download collections |
05:40
🔗
|
underscor |
Yeah, assuming he likes it |
05:40
🔗
|
underscor |
(ping SketchCow, so he sees it) |
05:41
🔗
|
underscor |
Also need to write hooks that will automatically do junk when items within a collection are updated |
05:41
🔗
|
underscor |
Need to see if they'll let me touch petabox code |
05:41
🔗
|
underscor |
;D |
05:44
🔗
|
SketchCow |
They won't. |
05:44
🔗
|
SketchCow |
But provide assistance. |
05:45
🔗
|
underscor |
Quick, the boss is here |
05:45
🔗
|
underscor |
Look busy! |
05:45
🔗
|
underscor |
SketchCow: Yeah, I know. Hank and BK are still sore about that incident in October |
05:45
🔗
|
underscor |
(rightfully so) |
05:46
🔗
|
underscor |
DFJustin: SketchCow: 8 down, 800 to go! http://hastebin.com/cowotigili.hs |
05:49
🔗
|
underscor |
closure: git-annex: /home/abuie/digibarn/.git/annex/tmp/remote_web_182_980_URL-s4619--http&c%%archive.org%download%DigibarnBruceDamerOnHowWilliamShatnerChangedTheWorldhistoryChannel%DigibarnBruceDamerOnHowWilliamShatnerChangedTheWorldhistoryChannel.thumbs%history-channel-shatner-digibarn-brucedamer__000390.jpg.log: openBinaryFile: invalid argument (File name too long) |
05:49
🔗
|
underscor |
:( |
05:52
🔗
|
closure |
ouch.. |
05:52
🔗
|
closure |
I will fix that tomorrow |
05:54
🔗
|
DFJustin |
I guess to really have ALLSHAREWARE you'll want http://www.archive.org/details/tucows as well |
05:54
🔗
|
DFJustin |
(the description on which is now out-of-date....) |
05:54
🔗
|
underscor |
closure: <3 |
05:54
🔗
|
underscor |
Thanks! |
05:54
🔗
|
underscor |
DFJustin: Yeah, I plan on doing it too |
05:55
🔗
|
underscor |
But once I have an automation workflow in place |
05:55
🔗
|
underscor |
Also, still wanting to hand-comb output for the time being |
05:55
🔗
|
underscor |
:) |
05:56
🔗
|
DFJustin |
they need to do another tucows pull too, "this just in...7 years ago" |
05:56
🔗
|
DFJustin |
the site is amazingly still up |
05:58
🔗
|
underscor |
wow |
05:58
🔗
|
underscor |
that's pretty impressive |
05:58
🔗
|
underscor |
I remember using tucows in like 2004 |
05:59
🔗
|
underscor |
back on my G3 AIO |
05:59
🔗
|
underscor |
:D |
05:59
🔗
|
underscor |
Fuckin' 10 years old |
05:59
🔗
|
underscor |
hahaha |
05:59
🔗
|
DFJustin |
I remember when "winsock software" actually meant something |
06:00
🔗
|
underscor |
:o |
06:00
🔗
|
underscor |
That was a while ago |
06:00
🔗
|
underscor |
;P |
06:00
🔗
|
DFJustin |
but I'm still a whippersnapper compared to some folks in here :) |
06:02
🔗
|
underscor |
Very true |
06:02
🔗
|
underscor |
Man, I remember playing this game |
06:02
🔗
|
underscor |
Damn, what was it called |
06:02
🔗
|
underscor |
It was like, you pretended you were in a museum |
06:02
🔗
|
underscor |
and there were all these puzzles and stuff you had to solve |
06:03
🔗
|
underscor |
I remember that in 3rd grade START (gifted education) on Windows 95 |
06:03
🔗
|
underscor |
Man, now I really want to know what it was called >:| |
06:05
🔗
|
DFJustin |
in 3rd grade we had apple IIs |
06:05
🔗
|
underscor |
We had apple IIs until 2nd grade |
06:05
🔗
|
closure |
underscor: fixed |
06:06
🔗
|
underscor |
and then we got celerons with windows 95 |
06:06
🔗
|
underscor |
closure: :D |
06:06
🔗
|
underscor |
We couldn't use "Batang" on those celerons |
06:06
🔗
|
underscor |
It would crash the computer |
06:06
🔗
|
underscor |
I remember that, too |
06:06
🔗
|
underscor |
God, all these memories |
06:07
🔗
|
underscor |
OMGOMGOMGOMGOMGOMG |
06:07
🔗
|
underscor |
I FOUND IT |
06:07
🔗
|
* |
underscor is so excited |
06:07
🔗
|
underscor |
http://www.abandonia.com/en/games/479/Museum+Madness.html |
06:07
🔗
|
closure |
is that a text adventure game? |
06:08
🔗
|
underscor |
Nope, windows 3.11 and 95 |
06:08
🔗
|
underscor |
Man, I'm gonna have to set up a win95 vm so I can play this <3 |
06:09
🔗
|
underscor |
closure: Thanks again for fixing that bug |
06:09
🔗
|
underscor |
Couldn't resist, eh? ;) |
06:09
🔗
|
underscor |
Wow, lots of changes! |
06:09
🔗
|
closure |
turned out to be easy, can just truncate and add a md5 for uniqueness |
06:09
🔗
|
underscor |
oic |
06:11
🔗
|
underscor |
http://www.amazon.com/Museum-Madness/dp/B0009U7CLQ |
06:11
🔗
|
underscor |
ha ha |
06:11
🔗
|
underscor |
prime eligible! |
06:12
🔗
|
underscor |
Customers buy this item with The Oregon Trail, 5th Edition by The Learning Company Windows 98 / Me / 95, Mac |
06:12
🔗
|
underscor |
Frequently Bought Together |
06:12
🔗
|
underscor |
Hehehehehe |
06:12
🔗
|
underscor |
I'm sure TONS of people have bought that combo lolololol |
06:13
🔗
|
underscor |
closure: Works like a charm, many thanks! |
06:54
🔗
|
Coderjoe |
underscor: so how big and unweildy is the git repo so far? |
06:54
🔗
|
Coderjoe |
wow |
06:54
🔗
|
Coderjoe |
70-01-08.us is kinda overloaded O_o |
07:03
🔗
|
underscor |
ha |
07:03
🔗
|
underscor |
Coderjoe: http://hastebin.com/totedewice.hs |
07:03
🔗
|
underscor |
Keep in mind that they're actually smaller than that, it's just in the middle of adding files |
07:03
🔗
|
underscor |
(re the du) |
07:08
🔗
|
Coderjoe |
seems like a lot of disk space for 1765 urls |
07:09
🔗
|
yipdw |
would repacking help? |
07:11
🔗
|
underscor |
Like I said, there are a bunch of temporary cruft laying around |
07:11
🔗
|
underscor |
s/are/is/ |
07:11
🔗
|
underscor |
They're all adding files, which takes up space until they're dropped |
07:12
🔗
|
yipdw |
PSA: Sleepytime tea + agave nectar = apotheosis of human creation |
08:19
🔗
|
Coderjoe |
closure: btw, when you're opening files for hashing in git-annex, are you opening them with the noatime option? |
08:19
🔗
|
Coderjoe |
http://kerneltrap.org/node/14148 |
08:22
🔗
|
cryptops1 |
someone here archived lachlan cranswick page, why can't i find it on the wiki under 'people' ? |
09:00
🔗
|
yipdw |
oparty |
09:01
🔗
|
SketchCow |
Hooray, I fixed the jamendo bug |
09:01
🔗
|
SketchCow |
Asked Emjirp to help me find ones that messed up. |
09:01
🔗
|
SketchCow |
Shouldn't be many. |
09:01
🔗
|
Coderjoe |
cryptops1: uh... because I forgot to add it to the wiki? |
09:03
🔗
|
Coderjoe |
other than possibly uploading it to batcave (and perhaps downloading it to home), I can no longer remember what I did with the resulting warc file |
09:04
🔗
|
Coderjoe |
ah. found it |
09:04
🔗
|
Coderjoe |
2.7G lachlan_cranswick/ |
09:07
🔗
|
cryptops1 |
np |
09:07
🔗
|
cryptops1 |
i was afraid it got thrown out |
09:08
🔗
|
Coderjoe |
and, iirc, there is a copy on batcave for SketchCow to push into archive.org at some point |
09:09
🔗
|
SketchCow |
I delete everything |
09:09
🔗
|
SketchCow |
Need more space for dwarf fortress 2012 |
09:10
🔗
|
Coderjoe |
World of Wanking 2012? |
09:10
🔗
|
Coderjoe |
with the new Hot Elf Babes expansion pack? |
09:13
🔗
|
ersi |
<@SketchCow> I delete |
09:13
🔗
|
ersi |
Haha, like that'd happen! |
09:59
🔗
|
chronomex |
mmm world of wanking |
10:07
🔗
|
SketchCow |
I am slamming 6,200 podcasts into archive.org. |
10:07
🔗
|
SketchCow |
http://www.archive.org/details/mypodcast-2020 |
10:26
🔗
|
SketchCow |
http://www.archive.org/details/mypodcast-zstalkshow |
10:26
🔗
|
SketchCow |
It's beginning. |
10:26
🔗
|
* |
SketchCow bows. |
10:38
🔗
|
* |
BlueMax laughs |
10:38
🔗
|
BlueMax |
Found this while browsing r/wtf on reddit |
10:38
🔗
|
BlueMax |
http://i.imgur.com/FW8xz.png |
10:41
🔗
|
SketchCow |
Here, BlueMax http://www.archive.org/details/mypodcast-dragonballradio |
10:42
🔗
|
BlueMax |
SketchCow: ? |
10:42
🔗
|
BlueMax |
What's this supposed to be |
10:42
🔗
|
SketchCow |
I'm rewarding you |
10:43
🔗
|
* |
BlueMax scratches his head |
10:43
🔗
|
nitro2k01 |
Over 9000 rewards |
10:47
🔗
|
SketchCow |
26 podcasts already up |
10:47
🔗
|
SketchCow |
BlueMax: Can you link me to the wtf |
10:48
🔗
|
BlueMax |
www.reddit.com/r/WTF/comments/pqrya/who_seriously_sits_and_writes_this_stuff/ |
10:48
🔗
|
ersi |
ahaha |
10:49
🔗
|
ersi |
and that links to this image with "Ariel's Wedding night" on textfiles.com: http://i.imgur.com/FW8xz.png |
10:49
🔗
|
BlueMax |
what ersi said |
10:51
🔗
|
BlueMax |
To think that page wouldn't have survived if it wasn't backed up |
10:51
🔗
|
SketchCow |
I've got 6 different scripts running to push these podcasts in. |
10:54
🔗
|
BlueMax |
And we should all be greatful |
11:42
🔗
|
godane |
mirroring blackhat.com |
11:45
🔗
|
kin37ik |
wait, what the hell? lol |
11:48
🔗
|
godane |
why is www.geocities.com.7z.302 is listed as WAVE file on archive.org |
11:48
🔗
|
godane |
http://www.archive.org/details/2009-archiveteam-geocities-part1 |
11:52
🔗
|
kin37ik |
is that the fixed archive? |
11:53
🔗
|
godane |
i don't know |
11:54
🔗
|
godane |
i think i need to get a bluray burner |
11:54
🔗
|
kin37ik |
okay, sure enough i grabbed geocities off the torrent wire, but i had to grab some files again because some archives were damaged, though im not sure which ones |
11:55
🔗
|
kin37ik |
not sure if it might be the same on archive.org or whether it's been appended |
11:59
🔗
|
Nemo_bis |
compare the checksums |
12:02
🔗
|
godane |
i'm just thinking of squashfs |
12:02
🔗
|
godane |
cause squashfs does hard links if files are the same |
12:04
🔗
|
godane |
i have full squashfs file of www.defcon.org that can be used host it locally on local lan |
12:20
🔗
|
SketchCow |
Archive.org isn't hot with some things. |
12:20
🔗
|
SketchCow |
.302 is a sound extension |
12:20
🔗
|
SketchCow |
But this isn't a sound file. |
12:21
🔗
|
godane |
thats what i thought |
12:28
🔗
|
SketchCow |
http://www.archive.org/details/mypodcast-beatmd |
12:28
🔗
|
SketchCow |
14 hours of sound mix! |
12:32
🔗
|
Nemo_bis |
All those file didn't have any license attached? |
12:40
🔗
|
godane |
looks like alot of .ppt files are 404 on blackhat.com |
12:40
🔗
|
godane |
for blackhat 2001 |
12:59
🔗
|
godane |
blackhat.com was not that big |
12:59
🔗
|
godane |
only 1.2gb |
12:59
🔗
|
godane |
defcon.org is like 3.8gb |
13:16
🔗
|
godane |
alot of .pdf didn't get download or are 404 on blackhat.com |
13:16
🔗
|
godane |
thats not good |
13:19
🔗
|
godane |
1485 broken links it looks like |
13:24
🔗
|
emijrp |
I heard that video talks contain hidden info using steganography. |
13:29
🔗
|
godane |
i have to redownload it |
13:29
🔗
|
godane |
stupid wget-warc delete everything i think in then try redownloaded it |
13:30
🔗
|
godane |
i thought it was only redownloaded files i had |
13:31
🔗
|
godane |
anyways i added a -o www.blackhat.com.log to the end of my command |
14:43
🔗
|
closure |
Coderjoe: for hashing git-annex uses sha256sum etc commands. So whatever they do. |
14:46
🔗
|
underscor |
closure: Whenever you have a few minutes, want to look this over and see if I did anything wrong? |
14:46
🔗
|
underscor |
https://github.com/ArchiveTeam/ia-digibarn |
14:53
🔗
|
closure |
underscor: yeah.. you need to push the git-annex branch too |
14:53
🔗
|
underscor |
0 2:53PM:abuie@abuie-dev:~/digibarn 23635 Ã git annex status |
14:53
🔗
|
underscor |
ae80d947-67dd-46e7-88be-1503f57cd03b -- here (DigiBarn (@ IA)) |
14:53
🔗
|
underscor |
semitrusted repositories: 1 |
14:53
🔗
|
underscor |
supported backends: SHA256 SHA1 SHA512 SHA224 SHA384 SHA256E SHA1E SHA512E SHA224E SHA384E WORM URL |
14:53
🔗
|
underscor |
supported remote types: git bup directory rsync web hook |
14:53
🔗
|
underscor |
trusted repositories: 0 |
14:53
🔗
|
underscor |
untrusted repositories: 1 |
14:53
🔗
|
underscor |
00000000-0000-0000-0000-000000000001 -- web |
14:53
🔗
|
underscor |
dead repositories: 0 |
14:53
🔗
|
underscor |
local annex keys: 0 |
14:53
🔗
|
underscor |
local annex size: 0 bytes |
14:54
🔗
|
underscor |
known annex keys: 2190 |
14:54
🔗
|
underscor |
known annex size: 12 gigabytes |
14:54
🔗
|
underscor |
backend usage: |
14:54
🔗
|
underscor |
URL: 2190 |
14:54
🔗
|
underscor |
closure: Ok, pushing now |
14:54
🔗
|
closure |
it should auto-push after the 1st time.. that's where git-annex keeps its info |
14:54
🔗
|
underscor |
So, after I git push -u origin git-annex once, it should auto-push from then on? |
14:55
🔗
|
closure |
that's been my experience |
14:55
🔗
|
underscor |
okay, cool |
14:55
🔗
|
underscor |
thanks! |
14:56
🔗
|
underscor |
Oh, wow, lots of objects in that branch |
14:56
🔗
|
underscor |
haha |
15:05
🔗
|
underscor |
70% compressing |
15:05
🔗
|
closure |
wow, that's more objects than I'd expect |
15:05
🔗
|
closure |
or an arm box? |
15:06
🔗
|
closure |
it'll have 2 files in the branch per file in the repo, plus some directories etc.. |
15:06
🔗
|
closure |
oh, you did individual addurls per file.. so it also has 2000 commits I guess |
15:43
🔗
|
closure |
closure: looks ok |
15:43
🔗
|
closure |
woah |
15:43
🔗
|
* |
closure core dumps |
15:50
🔗
|
Archivis2 |
closure: ~17k objects |
15:51
🔗
|
Archivis2 |
It's on an overworked nfsmount, so that doesn't help with IO |
15:51
🔗
|
closure |
yeah, so it's some unavoidable, but some of it can be improved |
15:51
🔗
|
godane |
how do you spider a website without download? |
15:52
🔗
|
Archivis2 |
closure: Still pretty damn impressive! |
15:52
🔗
|
Archivis2 |
:) |
15:52
🔗
|
Archivis2 |
I'm testing the recursive collection downloader to git-annex script I wrote this morning on http://archive.org/details/vectrex |
15:53
🔗
|
Archivis2 |
Nothing like being bored in statistics! |
15:53
🔗
|
* |
Archivis2 chuckles |
15:53
🔗
|
closure |
well, I'm thinking of adding an option --pathdepth=N , and it would take the last N parts of the path and use that for the filename. Then you could run one single git annex addurl and pass it all the urls in one go. This would be more efficient. |
15:54
🔗
|
Archivis2 |
That'd be pretty neat |
15:54
🔗
|
Archivis2 |
Although some of these are >1,000 files, so they probably would need to be batched smaller |
15:55
🔗
|
Archivis2 |
closure: It works! \o/ http://hastebin.com/siyiyinoki.hs |
15:55
🔗
|
Archivis2 |
(I purposely did a small collection to start with) |
15:55
🔗
|
closure |
hmm, command line length limits you mean? |
15:55
🔗
|
closure |
possibly a problem yes |
15:55
🔗
|
closure |
xargs it |
15:56
🔗
|
closure |
I do wonder if one repo per collection is the right granularity. You could put them all in one repo, might be more fun :) |
16:01
🔗
|
Archivis2 |
All IA items in one repo? |
16:01
🔗
|
Archivis2 |
I dunno how well it'd handle that |
16:02
🔗
|
closure |
all archiveteam items in one repo |
16:02
🔗
|
Archivis2 |
Oh, yes |
16:02
🔗
|
Archivis2 |
That's how it's going to be |
16:02
🔗
|
closure |
all IA would probably be insane |
16:03
🔗
|
Archivis2 |
vectrex and digibarn are collections similar to AT |
16:03
🔗
|
Archivis2 |
(number-of-items-wise) |
16:03
🔗
|
Archivis2 |
AT one is running, but it takes much longer because there is a large number of individual files |
16:04
🔗
|
closure |
ah, I see |
16:04
🔗
|
Archivis2 |
Especially in the geocities/yahoo video things, where they're split into smaller 7z.nnn |
16:04
🔗
|
closure |
let me write this feature and you can have much faster runtime |
16:04
🔗
|
Archivis2 |
\o/ |
16:06
🔗
|
Archivis2 |
https://github.com/ArchiveTeam/ia-vectrex |
16:06
🔗
|
Archivis2 |
git-annex branch still pushin |
16:06
🔗
|
Archivis2 |
g |
16:14
🔗
|
tef |
http://blog.archive.org/2012/02/15/want-to-help-build-a-distributed-web/ |
16:17
🔗
|
closure |
Just depth |
16:17
🔗
|
closure |
filter (not . null) $ split "/" fullurl |
16:17
🔗
|
closure |
fromend depth $ map escape $ |
16:17
🔗
|
closure |
| depth > 0 -> filesize $ join "/" $ |
16:17
🔗
|
closure |
oh yeah |
16:25
🔗
|
closure |
Archivis2: ok, --pathdepth pushed |
16:25
🔗
|
closure |
I'd be curious to see the comparison importing using it and xargs |
16:28
🔗
|
soultcer |
Sweet, I wanted to try git-annex for archiveteam today and now I come here and someoen already did all the work ;-) |
16:30
🔗
|
soultcer |
Are they checksummed? |
16:30
🔗
|
closure |
not the way he's doing it, it would need to download them all. but can be migrated to checksums later |
16:31
🔗
|
closure |
git annex migrate --backend=SHA256 |
16:31
🔗
|
soultcer |
The meta.xml files do contain checksums, maybe it is possible without downloading everything? |
16:31
🔗
|
closure |
oh, hmm, suppose |
16:32
🔗
|
soultcer |
Oh, not meta.xml, _files.xml |
16:32
🔗
|
soultcer |
Though it only carries md5 and sha1 sums, so you'd have to use the sha1 backend |
16:41
🔗
|
closure |
maybe something like git annex migrate --converter=xml-parsing.rb |
16:42
🔗
|
closure |
will think on it |
16:45
🔗
|
cryptops1 |
does anyone happen to have freenode's podcasts? |
16:45
🔗
|
cryptops1 |
the freenode network released some rather obscene podcasts around year 2006-2008? |
16:45
🔗
|
closure |
lol, really? |
16:45
🔗
|
cryptops1 |
yea they were pretty sweet! |
16:45
🔗
|
cryptops1 |
according to their descriptions |
16:45
🔗
|
closure |
was lilo in them? |
16:46
🔗
|
cryptops1 |
and they won't upload them for me since i'm life-banned on that network |
16:46
🔗
|
cryptops1 |
i don't know, i don't think so |
16:46
🔗
|
closure |
he was an old pal of mine, it would be good to hear his voice |
16:46
🔗
|
cryptops1 |
if he faked his death, you won't be hearing from him |
16:47
🔗
|
cryptops1 |
i always like to think people faked their death with a small chance of finding them |
16:47
🔗
|
cryptops1 |
but i've considered the possibility when my friends have 'died' |
16:47
🔗
|
cryptops1 |
anyways ... there really ARE freenode podcasts! |
16:47
🔗
|
cryptops1 |
i think the url is still up |
16:48
🔗
|
cryptops1 |
podcast.freenode.net |
16:48
🔗
|
cryptops1 |
i was wrong, they are from year 2009 |
17:02
🔗
|
balrog_ph |
cryptops1: Lol how did you get life banned? |
17:06
🔗
|
cryptops1 |
don't know, but i'm ban evading right now probably |
17:09
🔗
|
godane |
i'm getting all the '404' files from blackhat.com |
17:10
🔗
|
godane |
i did a spider to what i got of a local version of mirror of blackhat.com |
17:10
🔗
|
godane |
1496 files were missing |
17:12
🔗
|
godane |
there are still 404 errors in the list but i'm getting most of it |
17:34
🔗
|
cryptops1 |
one of the topics for a freenode postcast is 00:22:59 - We beg for money |
17:34
🔗
|
cryptops1 |
so maybe lilo was still around back then |
18:02
🔗
|
Archivis2 |
closure: Thanks! |
18:02
🔗
|
Archivis2 |
How do I use it? |
18:03
🔗
|
SketchCow |
Hey. |
18:06
🔗
|
Archivis2 |
Hi SketchCow |
18:06
🔗
|
Archivis2 |
How goes it? |
18:07
🔗
|
SketchCow |
Are you new or did you change your name? |
18:07
🔗
|
Archivis2 |
Oh, sorry |
18:07
🔗
|
underscor |
Let me go change the default for efnet |
18:08
🔗
|
underscor |
(I don't usually irc from my laptop, so all my settings aren't on here) |
18:11
🔗
|
closure |
underscor: you should be able to do something like: geturls | xargs git-annex addurl --fast --pathdepth=2 |
18:12
🔗
|
underscor |
where urls spits out the URL list |
18:12
🔗
|
closure |
yep |
18:13
🔗
|
underscor |
So, if I have example.com/dir1/file1.mp3 example.com/dir1/dir2/file2.mp3 and example.com/dir1/dir2/dir3/file3.mp3 will I end up with dir1/file1.mp3, dir1/dir2/file2.mp3 and dir1/dir2/dir3/file3.mp3? |
18:17
🔗
|
closure |
with pathdepth=2 yes |
18:17
🔗
|
closure |
um, no |
18:18
🔗
|
closure |
it currently takes the last 2 parts of the path |
18:18
🔗
|
closure |
maybe that should be --pathdepth=-2 and --pathdepth=2 should *skip* the first two parts? |
18:19
🔗
|
underscor |
That would be more useful |
18:19
🔗
|
underscor |
At least, in my usage scenario |
18:20
🔗
|
underscor |
Because I just don't want the ia6xxxxx.archive.org/16/items/ bit, but I want everything else |
18:20
🔗
|
underscor |
(which may be an arbitrary length |
18:20
🔗
|
underscor |
) |
18:26
🔗
|
swebb2 |
It turns out that I can't rsync to the badcave server anymore. Is this a temporary thing or has my access been removed? |
18:26
🔗
|
swebb2 |
s/badcave/batcave/ |
18:27
🔗
|
closure |
underscor: done |
18:28
🔗
|
closure |
| depth > 0 -> frombits $ drop depth |
18:28
🔗
|
closure |
| depth < 0 -> frombits $ reverse . take (negate depth) . reverse |
18:28
🔗
|
underscor |
schweeeeeeet! |
18:28
🔗
|
underscor |
swebb2: Batcave is offline, we're moving to fos/fortressofsolitude |
18:28
🔗
|
underscor |
Have to talk to SketchCow for a new module |
18:28
🔗
|
underscor |
(I dunno if he's doing them yet though) |
18:32
🔗
|
swebb2 |
ok. Can someone just email me my new creds or something when they're ready? |
18:41
🔗
|
underscor |
Poke SketchCow |
18:41
🔗
|
underscor |
He's the gatekeeper :) |
18:42
🔗
|
underscor |
Or just drop your email in here and we'll get it to him |
18:42
🔗
|
underscor |
s/email/email address/ |
18:44
🔗
|
SketchCow |
Whaaa |
18:45
🔗
|
SketchCow |
I just pinged him. |
18:46
🔗
|
SketchCow |
Basically, I'm having people verify with me which directories are done, so I can start pushing them along. |
18:47
🔗
|
underscor |
Okay, I see |
18:48
🔗
|
alard |
SketchCow: Any news about the umich data? |
18:48
🔗
|
SketchCow |
IT's syncing, but REALLY slowly. |
18:49
🔗
|
SketchCow |
It's only down to C. |
18:49
🔗
|
alard |
Oh, that's not fast. I'll be patient then. |
18:50
🔗
|
underscor |
SketchCow: How much space does fos have? |
18:50
🔗
|
underscor |
Just curious :D |
18:50
🔗
|
SketchCow |
I'm not letting you on fos. |
18:50
🔗
|
SketchCow |
I'm going to find you your own machine. |
18:50
🔗
|
SketchCow |
Right now, though I'm finding enormous slowdowns on fos and I am worried I am sharing a cancerous baremetal host. |
18:51
🔗
|
underscor |
Yeah, I was noticing that |
18:51
🔗
|
underscor |
(looking at the stats in the dom0) |
18:52
🔗
|
underscor |
SketchCow: Oh, okay. That works too, I suppose :D |
18:52
🔗
|
SketchCow |
The new machine is aprt of a new breed and I do not know the story. |
18:53
🔗
|
underscor |
Yeah, it's one of the new ganeti fai machines |
18:53
🔗
|
underscor |
(andy's been teaching me the infra) |
18:54
🔗
|
underscor |
What's strange is neither the VM nor the baremetal is spending a significant amount of time on iowait |
18:54
🔗
|
underscor |
We kept seeing that on the dev vm |
18:55
🔗
|
SketchCow |
Why aren't you asking andy for a machine? |
18:55
🔗
|
SketchCow |
That's all I'm going to do. |
18:55
🔗
|
underscor |
Oh, you're already running as cache=none, too |
18:55
🔗
|
underscor |
That's something I would have suggested to increase performance |
18:56
🔗
|
SketchCow |
I could also go "Why are you discussing internal archive.org infrastructure on a public channel on EFnet" but I gave up on that thread. |
18:56
🔗
|
underscor |
Well, I'm just some outside stranger |
18:56
🔗
|
underscor |
I figured you had more credance with him than I |
18:56
🔗
|
SketchCow |
I'm louder than you, yes. |
18:56
🔗
|
underscor |
Well, you're also an employee, unlike me |
18:56
🔗
|
underscor |
SketchCow: I haven't said anything that's not already publicly available knowledge |
18:57
🔗
|
SketchCow |
Tell yourself that. |
18:57
🔗
|
underscor |
But I'll quiet down about it then |
18:57
🔗
|
underscor |
Tell myself which? |
18:58
🔗
|
SketchCow |
Tell yourself that constant, consistent dumping of information on the operation of archive.org's machinery and methodology for its operation and remote manipulation will have no far-reaching consequences. |
18:59
🔗
|
SketchCow |
Also, have some pie |
19:01
🔗
|
underscor |
Alright, touche' |
19:01
🔗
|
underscor |
You're right, nothing good can possibly come from it. |
19:02
🔗
|
SketchCow |
You're young, and you're happy to be seeing the parts of the engineering that impresses you, but the ease of use that inevitably comes from some of the choices in the system, which date back to when literally 1 or 3 people had total knowledge of the environment and responsibility for it, which do not scale to univerally being known. |
19:02
🔗
|
SketchCow |
...are not good to drop. |
19:04
🔗
|
underscor |
I see. Thanks. |
19:04
🔗
|
SketchCow |
Also, I confess that the bathhouse wasn't clothing optional |
19:05
🔗
|
SketchCow |
And there is actually no "rub room" |
19:06
🔗
|
SketchCow |
I hit a steam room in Helsinki, those guys know how to run a steam room |
19:08
🔗
|
closure |
indeed. crazy find and their birch branch flagellation |
19:08
🔗
|
underscor |
haha |
19:08
🔗
|
closure |
finns even |
19:08
🔗
|
SketchCow |
Our finn host told us to do that |
19:08
🔗
|
SketchCow |
We go to the sauna, and the guy who runs it looks at us, and goes "... first time?" |
19:08
🔗
|
closure |
our finn host told us not to drink alcohol in the really hot room, or we could die. |
19:08
🔗
|
SketchCow |
"you don't need to do that." |
19:09
🔗
|
SketchCow |
Also, the snow thing was awesome |
19:09
🔗
|
SketchCow |
The guy who runs it has the demeanor of hourly motel clerk |
19:09
🔗
|
cryptops1 |
was there glory holes? |
19:09
🔗
|
closure |
oh, you went to the helsinki ice bar? |
19:09
🔗
|
SketchCow |
And when we were there, my lady has the steam room to herself |
19:10
🔗
|
SketchCow |
The ladies steam room |
19:10
🔗
|
SketchCow |
and he said it was fine for me to go up there |
19:10
🔗
|
SketchCow |
so we had a steam room to ourselves for a while |
19:10
🔗
|
closure |
tmi |
19:10
🔗
|
SketchCow |
No, not the ice bar |
19:10
🔗
|
SketchCow |
I mean the hanging out outside and rubbing snow on yourself thing |
19:11
🔗
|
closure |
yeah, in summer you just have to jump in the lake and swim as deep as possible, not as nice |
19:12
🔗
|
SketchCow |
http://www.flickr.com/photos/mirka23/6849416475/in/photosof-textfiles/ |
19:17
🔗
|
SketchCow |
I was told this was the best sauna, but I bet it was the most authentic |
19:17
🔗
|
SketchCow |
Wood-fired steam, etc. |
19:19
🔗
|
closure |
looks suspiciously like a train |
19:19
🔗
|
closure |
oh, slideshow |
20:33
🔗
|
closure |
underscor: how's it going? |
20:36
🔗
|
closure |
I have a new command pushed for you (or soultcer) |
20:37
🔗
|
closure |
git annex rekey --force file1 SHA1-sNNN--XXX file2 SHA1-sNNN-XXX ... |
20:37
🔗
|
chronomex |
sweet |
20:37
🔗
|
closure |
the NNN must be size in bytes, the NNN is the sha1 of course |
20:38
🔗
|
closure |
I'd recommend adding all the urls in one command, and then immedaitly rekeying them in one more command. That will be most efficient. |
20:40
🔗
|
soultcer |
One repository for all Archiveteam projects sounds pretty awesome |
20:43
🔗
|
closure |
er, the XXX is the sha1 , NNN is size |
20:44
🔗
|
Nemo_bis |
underscor, I don't understand, does having only your current machine actually prevent from doing something on the poor IA servers? :) |
20:44
🔗
|
Nemo_bis |
aka where are you uploading all this data to :p http://abuie-dev.us.archive.org:8088/mrtg/networkv2.html |
20:44
🔗
|
closure |
git annex rekey --force filea SHA1-s$asize--$asha fileb SHA1-s$bsize--$bsha ... |
20:47
🔗
|
Nemo_bis |
so long http://www.archive.org/details/mypodcast-dbandit |