| Time |
Nickname |
Message |
|
00:27
🔗
|
|
Start has joined #internetarchive.bak |
|
00:28
🔗
|
iabak-reg |
03registrar 05master cf23978 06other 10SHARD6/pubkeys registration of infinity on SHARD6 |
|
00:52
🔗
|
iabak-reg |
03registrar 05master 038c73d 06other 10SHARD7/pubkeys registration of kurtmclester on SHARD7 |
|
00:55
🔗
|
Kazzy |
shard5 done |
|
00:59
🔗
|
SketchCow |
https://soundcloud.com/renjith-vijay/fast-and-furious-7-get-low-ringtone |
|
00:59
🔗
|
SketchCow |
That plays every time we move shards. |
|
01:00
🔗
|
tpw_rules |
i've got 6TB down so far in general. how can i check progress on a specific shard? |
|
01:00
🔗
|
tpw_rules |
closure: ^ |
|
01:01
🔗
|
Kazzy |
tpw_rules: http://iabak.archiveteam.org/SHARD5.html |
|
01:01
🔗
|
Kazzy |
can substitute SHARD% for the shard you want to view |
|
01:01
🔗
|
tpw_rules |
i mean me specifically |
|
01:01
🔗
|
Kazzy |
SHARD5* |
|
01:15
🔗
|
closure |
yeah, we could make individual pages for each registered user. already have the graphs and data. |
|
01:19
🔗
|
tpw_rules |
what about just have it as part of git annex? |
|
01:20
🔗
|
closure |
tpw_rules: just run git-annex info |
|
01:23
🔗
|
tpw_rules |
oh duh |
|
01:24
🔗
|
tpw_rules |
why are some "unknown size"? |
|
01:24
🔗
|
closure |
xml files with dummy size 0 in the IA survey |
|
01:24
🔗
|
tpw_rules |
is the working tree the ones on your server or on disk size |
|
01:25
🔗
|
closure |
that's the total files, local annex is on your disk |
|
01:25
🔗
|
tpw_rules |
ok |
|
01:26
🔗
|
tpw_rules |
okay so i have about a third of shard6 |
|
01:26
🔗
|
tpw_rules |
gonna need to add more disk soon |
|
01:30
🔗
|
db48x |
tpw_rules: what's the UUID of your copy of the shard? |
|
01:32
🔗
|
db48x |
cd shard5; git config --get annex.uuid |
|
01:32
🔗
|
|
primus104 has quit IRC (Leaving.) |
|
01:34
🔗
|
|
BotOfWar has quit IRC (Quit: left4dead) |
|
01:35
🔗
|
tpw_rules |
shard5: ada4868b-f969-40ba-ae7e-d58a20f1c477 shard6: a0c7d62a-53ec-4641-8ecf-af122b2cd5da shard1: 6ca85911-2e94-49b6-bbdc-cb1be080e300 |
|
01:35
🔗
|
tpw_rules |
i'm talking about shard6 |
|
01:35
🔗
|
|
VADemon has joined #internetarchive.bak |
|
01:35
🔗
|
tpw_rules |
can you remove all the ones that claim to be twatson52 that aren't those? i made a bunch of ones that got blown up in various ways |
|
01:36
🔗
|
tpw_rules |
they all have the same email |
|
01:36
🔗
|
db48x |
http://iabak.archiveteam.org:8080/render/?width=1060&height=733&_salt=1432604162.838&target=iabak.shardstats.leaderboard.a0c7d62a-53ec-4641-8ecf-af122b2cd5da.shard6 |
|
01:37
🔗
|
db48x |
http://iabak.archiveteam.org:8080/render/?width=1060&height=733&_salt=1432604200.538&target=iabak.shardstats.leaderboard.a0c7d62a-53ec-4641-8ecf-af122b2cd5da.shard6&from=-1weeks |
|
01:37
🔗
|
db48x |
pretty nice slope there |
|
01:38
🔗
|
db48x |
you can use iabak.archiveteam.org:8080 to look at all the different stats we collect |
|
01:38
🔗
|
db48x |
gives you a nice browser and graph editor |
|
01:38
🔗
|
tpw_rules |
is there any reason the line is all dashy? |
|
01:39
🔗
|
db48x |
we only check in once per hour |
|
01:39
🔗
|
db48x |
you can use the graph editor to make it join up the dots |
|
01:39
🔗
|
tpw_rules |
but shouldn't it connect the dots? i get the stairstep but it looks like it was done with a dashed line |
|
01:39
🔗
|
tpw_rules |
ah |
|
01:40
🔗
|
db48x |
it doesn't know a priori what a gap in the data means |
|
01:40
🔗
|
db48x |
in this case it means a missed sample; it could just as easily mean a zero |
|
02:08
🔗
|
tpw_rules |
has anybody else fiddled around with union filesystems for storing all the data? |
|
02:09
🔗
|
db48x |
I use ZFS |
|
02:09
🔗
|
trs80 |
what's the goal of using a union fs? |
|
02:09
🔗
|
tpw_rules |
i'm coming at it from the point of having piles of hard drives that could be used |
|
02:09
🔗
|
tpw_rules |
basically be able to have a failure of one drive not affect all the data stored |
|
02:09
🔗
|
db48x |
yea |
|
02:10
🔗
|
db48x |
ZFS allows striping/mirroring/raid across disks |
|
02:10
🔗
|
tpw_rules |
and, more importantly, a failure of one drive not make the other data unusable like it would in a RAID 0 |
|
02:11
🔗
|
db48x |
true |
|
02:11
🔗
|
db48x |
I prefer raid though, so that the occasional error can be corrected |
|
02:11
🔗
|
tpw_rules |
but otoh i'm thinking about the idea of writing a disk manager that would create a new repo on each disk and manage filling them up and fscking as plugged in |
|
02:11
🔗
|
tpw_rules |
so i don't need 45 drive enclosures and usb ports |
|
02:11
🔗
|
db48x |
:) |
|
02:12
🔗
|
db48x |
we should have a way for the iabak client to help that out |
|
02:12
🔗
|
tpw_rules |
but then i have to actually swap drives around and i'm lazy |
|
02:13
🔗
|
tpw_rules |
well how in the hands of the masses do you want this to be? i know several people who would be happy to help if it were just something they kept running on windows in the tray |
|
02:14
🔗
|
tpw_rules |
and i really don't think making it all shell scripts is condusive to that ideal |
|
02:15
🔗
|
db48x |
very true |
|
02:16
🔗
|
sep332 |
With multiple disks you have to make sure that they're not all getting the same data |
|
02:17
🔗
|
tpw_rules |
also note that if i say i have an idea, you can say "shut up and make it" because i could. but here goes. have a config file of archive destinations. for each, give it a name and directory. in each directory, put a file with the name and a uuid |
|
02:17
🔗
|
sep332 |
No good having 3 copies of a file sitting next to each other in one drawer |
|
02:17
🔗
|
tpw_rules |
this way you can have the directory be a mount point (and the same for multiple desinations) and the file can be used to determine if a disk is mounted/which one is there |
|
02:17
🔗
|
tpw_rules |
sep332: oh yeah, that's important |
|
02:18
🔗
|
db48x |
sep332: yea, git annex can handle that if we tell it to |
|
02:18
🔗
|
tpw_rules |
though i get the approach of building this from common tools so it can be rebuilt in the event of thermonuclear meltdown, i think it limits options a lot |
|
02:19
🔗
|
tpw_rules |
but this program could also read the config and determine which ones need to be checked and ask for the appropriate one to show up at the directory |
|
02:19
🔗
|
tpw_rules |
so it could easily support multiple drives that are swapped out or multiple locations on one computer |
|
02:20
🔗
|
tpw_rules |
also perhaps the option to download locally so it can fsck one drive and later copy files to another when it comes back |
|
02:22
🔗
|
db48x |
we can simplify that more, even |
|
02:23
🔗
|
db48x |
git annex repositories already have their uuid |
|
02:23
🔗
|
tpw_rules |
can you scan a directory for a git repo easily? |
|
02:23
🔗
|
db48x |
yes |
|
02:23
🔗
|
tpw_rules |
recursively? |
|
02:23
🔗
|
tpw_rules |
but yeah. just read the .git/whatever in the pointed to directory to make sure |
|
02:23
🔗
|
db48x |
so if the iabak script listened for devices being added (udev, on linux) and found the repositories on them, it could fsck them |
|
02:24
🔗
|
tpw_rules |
ignore my previous two questions |
|
02:24
🔗
|
db48x |
yep |
|
02:24
🔗
|
tpw_rules |
does git exist for windows without cygwin? |
|
02:24
🔗
|
tpw_rules |
and could git annex? |
|
02:24
🔗
|
db48x |
no |
|
02:25
🔗
|
tpw_rules |
is it possible to bundle cygwin into an installer? |
|
02:25
🔗
|
db48x |
sure |
|
02:25
🔗
|
db48x |
git already does that |
|
02:25
🔗
|
tpw_rules |
i have some experience with windows gui programming (python/pyqt so it could even be cross-platform) and bundle all that stuff together |
|
02:25
🔗
|
db48x |
excellent :) |
|
02:27
🔗
|
tpw_rules |
now this is where it gets shaky: how do we tie multiple repos on multiple computers to one name? that would be necessary to prevent duplication. i have a friend with a stack of six or so laptops; they should all know the files the others have |
|
02:27
🔗
|
db48x |
yes, each repository knows which other repsotories have each file |
|
02:28
🔗
|
SketchCow |
Yeah, this is git-annex's job |
|
02:28
🔗
|
tpw_rules |
so we would have some association of the sets of repositories to an account and git annex can query those in the set? |
|
02:28
🔗
|
db48x |
mostly |
|
02:29
🔗
|
tpw_rules |
i mean we don't want two copies of a file three feet from each other, but having two copies three states from each other is good |
|
02:29
🔗
|
db48x |
tpw_rules: yes |
|
02:29
🔗
|
|
VADemon has quit IRC (Quit: left4dead) |
|
02:29
🔗
|
SketchCow |
I suspect we're going to run into a LITTLE of that no matter what, because people will want to "help" |
|
02:29
🔗
|
db48x |
theres a couple of ways to do it |
|
02:30
🔗
|
db48x |
there's a git-annex feature coming down the line at some point which we could rely on |
|
02:30
🔗
|
tpw_rules |
SketchCow: sure, but i wasn't sure how we automatically prevented the first and allowed the second |
|
02:30
🔗
|
db48x |
or iabak could simply say git annex get --not --copies 4 --not in otherrepo1 --not --in otherrepo2... |
|
02:31
🔗
|
tpw_rules |
that list would only update when a sync happened though |
|
02:31
🔗
|
tpw_rules |
might have to tune that in practice |
|
02:31
🔗
|
db48x |
the git annex feature is called balanced preferred content: http://git-annex.branchable.com/design/balanced_preferred_content/ |
|
02:32
🔗
|
db48x |
if the disk is offline, then the last sync for it has the must up-to-date information already :) |
|
02:32
🔗
|
tpw_rules |
well this could also support n drives simultaneously too |
|
02:33
🔗
|
db48x |
simplest to put a single repository on each of them |
|
02:33
🔗
|
tpw_rules |
i was thinking more across multiple computers. but the balanced thing would sovle that problem |
|
02:34
🔗
|
db48x |
both ways of doing it will work for n computers and n drives, and n drives on each of m computers ;) |
|
02:34
🔗
|
tpw_rules |
but which will be the most perfect |
|
02:35
🔗
|
db48x |
:) |
|
02:36
🔗
|
tpw_rules |
i'd be happy to write a gui but i don't know enough about git annex and bash to do the script. i feel we should move it to python or something |
|
02:36
🔗
|
tpw_rules |
or haskell :P |
|
02:36
🔗
|
db48x |
haskell would be fun |
|
02:36
🔗
|
db48x |
I don't know haskell very well yet; it'd be fun to learn |
|
02:36
🔗
|
tpw_rules |
i need to learn haskell |
|
02:37
🔗
|
tpw_rules |
i have worked with functionaly programming languages, but nothing truly functional |
|
02:44
🔗
|
SketchCow |
closure: Seems like I need to start giving you more collections |
|
02:44
🔗
|
SketchCow |
And we are getting to the point where we need to do a sanity check to make sure a collection isn't already being backed up |
|
02:46
🔗
|
closure |
I have that sanity check in place actually |
|
02:46
🔗
|
closure |
http://iabak.archiveteam.org/client/f9601d3062715f39f6290547fbaf14b3e6c2b4fb.html |
|
02:47
🔗
|
SketchCow |
Great |
|
02:50
🔗
|
iabak-reg |
03registrar 05master da43505 06other 10SHARD6/pubkeys registration of kurtmclester on SHARD6 |
|
02:56
🔗
|
SketchCow |
Shard6 is filling in nicely. |
|
04:03
🔗
|
tpw_rules |
maybe we don't need all commits in the channel :P |
|
04:53
🔗
|
SketchCow |
Yes. We. Do. |
|
04:55
🔗
|
tpw_rules |
okay okay okay |
|
04:57
🔗
|
tpw_rules |
/media/iabak/disk2/IA.BAK;/media/iabak/disk3/IA.BAK;/media/iabak/disk4/IA.BAK 8.1T 5.9T 1.8T 77% /home/thomas/iabak/IA.BAK |
|
04:57
🔗
|
SketchCow |
This is what we PLAYYYY FORRRRR |
|
04:57
🔗
|
tpw_rules |
fourtunately i have 3x 3TB disks that i'm not using right now |
|
04:58
🔗
|
tpw_rules |
anyway, it is time to say good night and let the datums flow in |
|
05:03
🔗
|
iabak-reg |
03registrar 05master a2a9eb2 06other 10SHARD6/pubkeys registration of kevin on SHARD6 |
|
06:00
🔗
|
|
zottelbey has joined #internetarchive.bak |
|
06:37
🔗
|
iabak-reg |
03registrar 05master 9e6e2ff 06other 10SHARD6/pubkeys registration of bas+at on SHARD6 |
|
07:48
🔗
|
Senji |
tpw: http://iabak.archiveteam.org/stats/SHARD5.leaderboard-raw seems to think you have 8T 8T 8T 1.5T |
|
07:51
🔗
|
Senji |
Err, no, divite all those numbers by 10, I can'tmath :) |
|
07:51
🔗
|
Senji |
I'll just go tback to bed, clearly I'm not awake yet |
|
07:53
🔗
|
lhobas_ |
new stats pages per user are really nice :) |
|
07:55
🔗
|
lhobas_ |
just noticed the cleanup function in the iabak script doesn't work on OS X (more stupid Mac-only glitches I assume) - https://github.com/ArchiveTeam/IA.BAK/blob/a420ad/iabak-helper#L282 throws "No such file or directory" (pid does exist, statement should eval to true) |
|
07:56
🔗
|
|
ivan` has joined #internetarchive.bak |
|
07:56
🔗
|
lhobas_ |
#L285 in that file seems off to me, think the file is supposed to be rm'ed, not the pid-number right? |
|
07:56
🔗
|
|
primus104 has joined #internetarchive.bak |
|
08:03
🔗
|
garyrh |
https://news.ycombinator.com/item?id=9602868 |
|
08:16
🔗
|
ivan` |
I would like to nominate https://archive.org/details/archiveteam_greader for distributed archival because it's got 8TB of compressed text, a lot from dead blogs that are nowhere else |
|
08:16
🔗
|
ivan` |
the Directory and Stats are unimportant and omitting them saves ~800GB |
|
08:16
🔗
|
ivan` |
I was planning on dumping it into my Google Drive or onto external drives but never got around to either but will maybe try later |
|
08:19
🔗
|
iabak-reg |
03registrar 05master 37605fb 06other 10SHARD6/pubkeys registration of cyrus on SHARD6 |
|
08:21
🔗
|
iabak-reg |
03registrar 05master 5463225 06other 10SHARD6/pubkeys registration of peter on SHARD6 |
|
08:26
🔗
|
|
Start has quit IRC (Read error: Connection reset by peer) |
|
08:26
🔗
|
|
Start_ has joined #internetarchive.bak |
|
08:28
🔗
|
|
Cyrus has joined #internetarchive.bak |
|
08:28
🔗
|
Senji |
Emcy: I'd say "2 sheets" rather than "4 or more" :) |
|
08:28
🔗
|
Senji |
Bah, mischat :) |
|
08:40
🔗
|
iabak-reg |
03registrar 05master d422268 06other 10SHARD6/pubkeys registration of antoine on SHARD6 |
|
09:17
🔗
|
SketchCow |
This project just got mentioned on hackernews. |
|
09:17
🔗
|
SketchCow |
Might cause a run on clients. |
|
09:17
🔗
|
SketchCow |
Or whiners. |
|
09:17
🔗
|
SketchCow |
Or whiny clients |
|
09:17
🔗
|
SketchCow |
Or client whiners |
|
09:19
🔗
|
iabak-reg |
03registrar 05master c394c48 06other 10SHARD6/pubkeys registration of koos303 on SHARD6 |
|
09:21
🔗
|
|
atomotic has joined #internetarchive.bak |
|
09:23
🔗
|
garyrh |
All four plus extra. |
|
09:25
🔗
|
|
atomotic has quit IRC (Client Quit) |
|
09:26
🔗
|
SketchCow |
We are probably going to start getting into the light realm of bad actors. |
|
09:26
🔗
|
SketchCow |
We'll see how we handle it. |
|
09:28
🔗
|
iabak-reg |
03registrar 05master f586bb7 06other 10SHARD6/pubkeys registration of info on SHARD6 |
|
09:36
🔗
|
|
Start_ has quit IRC (Read error: Connection reset by peer) |
|
09:36
🔗
|
|
lufix has joined #internetarchive.bak |
|
09:37
🔗
|
|
Start has joined #internetarchive.bak |
|
10:05
🔗
|
iabak-reg |
03registrar 05master 40f2c21 06other 10SHARD6/pubkeys registration of bdupray on SHARD6 |
|
11:03
🔗
|
|
lufix has quit IRC (Ping timeout: 240 seconds) |
|
11:03
🔗
|
iabak-reg |
03registrar 05master 9ebe0e8 06other 10SHARD6/pubkeys registration of alex_online78532 on SHARD6 |
|
11:10
🔗
|
|
lufix has joined #internetarchive.bak |
|
11:10
🔗
|
|
hendi has joined #internetarchive.bak |
|
11:27
🔗
|
hendi |
is there a way to set a nicer name for my account? currently I'm named "info"? |
|
11:28
🔗
|
Senji |
I believe there is an intention to allow nicknames. Currently it's using the bit before the @ in your email address |
|
11:28
🔗
|
hendi |
alright, thanks; I'll keep an eye out for that functionality then |
|
11:30
🔗
|
iabak-reg |
03registrar 05master 744b699 06other 10SHARD6/pubkeys registration of andrei.zbikowski on SHARD6 |
|
11:57
🔗
|
|
atomotic has joined #internetarchive.bak |
|
12:34
🔗
|
iabak-reg |
03registrar 05master d785a5d 06other 10SHARD6/pubkeys registration of olliejudge on SHARD6 |
|
12:39
🔗
|
|
beardicus has joined #internetarchive.bak |
|
12:40
🔗
|
beardicus |
hello, meat popsicles. |
|
12:40
🔗
|
beardicus |
i think i need to register. |
|
12:41
🔗
|
beardicus |
finally got a chance to update my scripts this morning... should be fscking shard1 right now. |
|
12:41
🔗
|
beardicus |
but i am "out of touch" |
|
12:41
🔗
|
ppiixx |
beardicus: i think you just need to run change-email in the iabak dir |
|
12:42
🔗
|
beardicus |
hmm. i now see `register-helper.pl`... let's see what that does. |
|
12:44
🔗
|
iabak-reg |
03registrar 05master a378f40 06other 10SHARD1/pubkeys registration of brian on SHARD1 |
|
12:44
🔗
|
iabak-reg |
03registrar 05master dde25de 06other 10SHARD2/pubkeys registration of brian on SHARD2 |
|
12:45
🔗
|
Senji |
That seems to have worked |
|
12:45
🔗
|
beardicus |
that was change-email that did it. thanks ppiixx |
|
12:46
🔗
|
beardicus |
noting that prompt-email did nothing. |
|
12:46
🔗
|
beardicus |
also noting that my system has neither systemd nor cron, so the script is a little complainy and i'll have to figure that out. |
|
12:47
🔗
|
Senji |
No cron! |
|
12:47
🔗
|
beardicus |
it's a synology nas. |
|
12:47
🔗
|
beardicus |
there's a gui thingy for running tasks though, i think. |
|
12:48
🔗
|
beardicus |
"Task Scheduler" yay. |
|
12:49
🔗
|
beardicus |
assuming the corrrect periodic command to blip is iabak-cronjob? daily? |
|
12:50
🔗
|
ppiixx |
yep |
|
12:50
🔗
|
|
sankin has joined #internetarchive.bak |
|
12:51
🔗
|
iabak-reg |
03registrar 05master b95aab9 06other 10SHARD6/pubkeys registration of olliejudge on SHARD6 |
|
12:56
🔗
|
lufix |
beardicus: My synology nas has cron, I believe? |
|
12:57
🔗
|
beardicus |
hmm. crond does exist. no crontab though. |
|
12:59
🔗
|
iabak-reg |
03registrar 05master 69747f3 06other 10SHARD6/pubkeys registration of atrus6 on SHARD6 |
|
13:30
🔗
|
lufix |
beardicus: Ah, I see :) http://www.multigesture.net/articles/how-to-use-cron-on-a-synology-nas/ |
|
13:30
🔗
|
lufix |
Might help |
|
13:31
🔗
|
beardicus |
yeah. apparently you need to use tabs between fields too. |
|
13:48
🔗
|
iabak-reg |
03registrar 05master e1e0acf 06other 10SHARD6/pubkeys registration of jon archive.org on SHARD6 |
|
13:50
🔗
|
iabak-reg |
03registrar 05master 629d09d 06other 10SHARD6/pubkeys registration of mail on SHARD6 |
|
13:55
🔗
|
|
Start has quit IRC (Disconnected.) |
|
14:18
🔗
|
iabak-reg |
03registrar 05master f8bf3c8 06other 10SHARD6/pubkeys registration of chiploaded on SHARD6 |
|
14:34
🔗
|
|
zottelbey has quit IRC (Ping timeout: 512 seconds) |
|
14:41
🔗
|
|
Start has joined #internetarchive.bak |
|
14:46
🔗
|
|
ohhdemgir has joined #internetarchive.bak |
|
14:52
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
|
14:55
🔗
|
iabak-reg |
03registrar 05master 09b57ac 06other 10SHARD6/pubkeys registration of moritz.steiner on SHARD6 |
|
15:17
🔗
|
|
zottelbey has joined #internetarchive.bak |
|
15:19
🔗
|
iabak-reg |
03registrar 05master 3988cd2 06other 10SHARD6/pubkeys registration of mariusz on SHARD6 |
|
15:20
🔗
|
|
beardicus has quit IRC (Sleep.) |
|
15:20
🔗
|
|
primus104 has quit IRC (Leaving.) |
|
15:28
🔗
|
iabak-reg |
03registrar 05master e5f4b99 06other 10SHARD6/pubkeys registration of iabackup on SHARD6 |
|
15:45
🔗
|
sep332 |
I second ivan`'s nomination of the google reader archive, even though the files are huge |
|
15:50
🔗
|
|
Start has quit IRC (Disconnected.) |
|
15:57
🔗
|
|
Start has joined #internetarchive.bak |
|
16:00
🔗
|
|
scatman has joined #internetarchive.bak |
|
16:01
🔗
|
|
Start has quit IRC (Client Quit) |
|
16:12
🔗
|
|
mariusz has joined #internetarchive.bak |
|
16:13
🔗
|
|
Zero_Dogg has joined #internetarchive.bak |
|
16:14
🔗
|
iabak-reg |
03registrar 05master 91c2b2a 06other 10SHARD6/pubkeys registration of archive.org on SHARD6 |
|
16:14
🔗
|
mariusz |
Hi. How can I register myself? :) |
|
16:16
🔗
|
Zero_Dogg |
Your install-git-annex is a bit stupid, always defaulting to i386. There are standalone tarballs for arm too, that works on raspberry pi (which I'd be using if I set it up) |
|
16:16
🔗
|
|
Start has joined #internetarchive.bak |
|
16:18
🔗
|
Zero_Dogg |
https://github.com/zerodogg/scriptbucket/blob/master/gitannex-install#L52-L60 is an example of the logic needed |
|
16:23
🔗
|
yipdw |
do submit patches, we all run on i386/x86-64 and therefore haven't had a need to generalize |
|
16:25
🔗
|
Zero_Dogg |
I will :) |
|
16:26
🔗
|
Zero_Dogg |
Got some spare space that I might be able to use for this, but it's on a raspi server |
|
16:26
🔗
|
SketchCow |
Zero_Dogg: So you have criticism and can't donate space! |
|
16:26
🔗
|
SketchCow |
You... you came from Hackernews |
|
16:27
🔗
|
Zero_Dogg |
does it require much cpu after the whole thing is downloaded (ie. does it git annex fsck, much)? |
|
16:27
🔗
|
Zero_Dogg |
SketchCow: lol |
|
16:28
🔗
|
Zero_Dogg |
SketchCow: I came from your blog, actually :p |
|
16:28
🔗
|
SketchCow |
Oh, THAT dump |
|
16:28
🔗
|
Zero_Dogg |
hah |
|
16:30
🔗
|
yipdw |
http://upload.wikimedia.org/wikipedia/commons/5/55/Creature_from_the_Black_Lagoon_poster.jpg |
|
16:39
🔗
|
SketchCow |
I will say, the Dogg is right in one regard - we should come up with some FAQ/information on how system intensive the ongoing holding of the data is. |
|
16:43
🔗
|
Zero_Dogg |
See? All nice and constructive, complete with pull request. Not hackernewsy at all |
|
16:45
🔗
|
|
Start has quit IRC (Disconnected.) |
|
16:46
🔗
|
SketchCow |
That's the way we like it. |
|
16:58
🔗
|
|
Lord has joined #internetarchive.bak |
|
16:58
🔗
|
Lord |
hello |
|
16:59
🔗
|
Lord |
i'm quite interested in this project (backink up the web backup :-) ) |
|
17:06
🔗
|
Lord |
i launched iabak, it downloaded some files and it failed |
|
17:06
🔗
|
Lord |
i created a user without home so the script failed |
|
17:06
🔗
|
Lord |
(maybe this info interest you) |
|
17:08
🔗
|
Lord |
i think i'll face another problem : the script is downloading gitannex i386 but my gentoo doesn't have multilib support |
|
17:08
🔗
|
|
Beardface has joined #internetarchive.bak |
|
17:09
🔗
|
iabak-reg |
03registrar 05master 256141c 06other 10SHARD6/pubkeys registration of lord-ia on SHARD6 |
|
17:10
🔗
|
Lord |
here i am :-) |
|
17:10
🔗
|
Lord |
it works |
|
17:10
🔗
|
Lord |
(with lots of setlocale errors) |
|
17:14
🔗
|
sep332 |
can git-annex be configured to use a set amount of space? like 2TB? |
|
17:19
🔗
|
mariusz |
sep332: git config annex.diskreserve 2000GB |
|
17:20
🔗
|
sep332 |
that's not how much space it *won't* use? |
|
17:23
🔗
|
|
primus104 has joined #internetarchive.bak |
|
17:26
🔗
|
mariusz |
sep332: you're right:) |
|
17:26
🔗
|
mariusz |
sep332: sorry |
|
17:26
🔗
|
sep332 |
np. it's a cool idea just not what i'm looking for |
|
17:29
🔗
|
SketchCow |
The new clients are cuasing some partying. |
|
17:30
🔗
|
iabak-reg |
03registrar 05master 3de6df9 06other 10SHARD6/pubkeys registration of dylan.barlett on SHARD6 |
|
17:33
🔗
|
tpw_rules |
is that a problem? i'll bring pizza |
|
17:33
🔗
|
tpw_rules |
i'm getting an error trying to sync to shard 6: error: Ref refs/heads/synced/git-annex is at 8c3a1a32ad19cc72f8429d7078dce8e9bc7e9e67 but expected e606f7f0f586c8f1504bd49cccb48d81dfa0a873 |
|
17:35
🔗
|
SketchCow |
No, it's not causing a problem at all. Just watching the activity. |
|
17:35
🔗
|
SketchCow |
We also are getting dilletantes, which is good, because that's a worthwhile experiment. |
|
17:36
🔗
|
SketchCow |
(People joining to fuck around and see what it does, then going "well that was fun" and disappearing, likely already, but certainly within the 2 week/4 week period) |
|
17:38
🔗
|
SketchCow |
My theory is this will just cause a bunch of 0.00 clients, since people are unlikely to go "let's see what it does.... DOWNLOAD A TERABYTE" |
|
17:39
🔗
|
iabak-reg |
03registrar 05master b29843a 06other 10SHARD6/pubkeys registration of frozenbeardme on SHARD6 |
|
17:42
🔗
|
iabak-reg |
03registrar 05master 5e39162 06other 10SHARD6/pubkeys registration of ryan on SHARD6 |
|
17:46
🔗
|
Beardface |
it works! |
|
17:48
🔗
|
SketchCow |
That's what we hope! |
|
17:48
🔗
|
SketchCow |
How much space you got, Beardface! |
|
17:48
🔗
|
* |
SketchCow rubs hands like Mr. Burns |
|
17:48
🔗
|
Beardface |
~1T so your last comment is kind of relevant, heh |
|
17:49
🔗
|
hendi |
I currently have 1.5TB and think about adding some more |
|
17:50
🔗
|
hendi |
should I do RAID1, or go without RAID, and just redownload when a drive fails? |
|
17:50
🔗
|
|
Start has joined #internetarchive.bak |
|
17:50
🔗
|
Beardface |
it checks for systemd to install a service, if not found it exits.. intentional? (when you start it again it installs a cron instead) |
|
17:52
🔗
|
|
primus104 has quit IRC (Leaving.) |
|
17:53
🔗
|
DFJustin |
hendi: there is already redundancy in the iabak software so that multiple people will get the same file, so local RAID is unnecessary |
|
17:54
🔗
|
SketchCow |
agree with DFJustin - it's wasted space, unless you personally have an interest in a collection in a way you are making it available elsewhere. |
|
17:58
🔗
|
hendi |
great, thank you |
|
17:58
🔗
|
hendi |
expect at least 15TB from me, then |
|
18:01
🔗
|
SketchCow |
Fantastic. |
|
18:01
🔗
|
SketchCow |
That'll help a lot. |
|
18:02
🔗
|
mariusz |
I plan on running iabak on more than one computer. Is the software smart enough to send me a different set of data to each one? |
|
18:02
🔗
|
tpw_rules |
not yet |
|
18:02
🔗
|
tpw_rules |
hendi: i've used mhddfs to attach a bunch of drives into one filesystem |
|
18:02
🔗
|
tpw_rules |
the advantage being that if one drive breaks, it doesn't take everything |
|
18:03
🔗
|
SketchCow |
mariusz: It's a worthwhile feature going forward for closure and db48x to consider, where machines are called buddies and they're treated as one machine. |
|
18:04
🔗
|
mariusz |
Yeah,tThat would be great.. |
|
18:05
🔗
|
SketchCow |
This project keeps coming up with new feature-adds |
|
18:06
🔗
|
mariusz |
another question. I have about 50 older drives ranging from 80GB to 1TB that I could manually plug-in once every few weeks. Is "cold storage" supported? If yes, any info how to go about this? |
|
18:06
🔗
|
SketchCow |
Somebody in 4 years is going to go "Man, this closure guy thought of EVERYTHING...." |
|
18:06
🔗
|
SketchCow |
mariusz: Not yet, in any meaningful way. |
|
18:06
🔗
|
SketchCow |
I should say it's supported in git-annex, but we're being simple... for now. |
|
18:06
🔗
|
tpw_rules |
mariusz: i solved that problem by getting a bunch of extremely cheap usb enclosures and attaching them as one |
|
18:06
🔗
|
SketchCow |
Because of aforementioned discovery of "buddy" feature and similar features. |
|
18:07
🔗
|
tpw_rules |
oh i had another idea: be able to set up an IPC socket that git annex will request downloads from so we can use something other than wget. i was thinking a pretty gui |
|
18:08
🔗
|
tpw_rules |
or even just a download command |
|
18:09
🔗
|
hendi |
tpw_rules, thanks for the hint, I'll have a look at mhddfs! |
|
18:09
🔗
|
tpw_rules |
get 0.1.38 btw, the later version is a bit crashy. it's a union fs, but it supports writes too |
|
18:13
🔗
|
DFJustin |
I think it does support cold storage as long as you check in at least once a month |
|
18:13
🔗
|
mariusz |
another idea - sneakerneting the data. i.e get a beer and copy your shard :) |
|
18:14
🔗
|
SketchCow |
It's a thought down the line. |
|
18:15
🔗
|
SketchCow |
There's a second/third/fourth wave of approach as we hit the upper limits of just scooping people out and into the project. |
|
18:15
🔗
|
SketchCow |
But it's still holding up for people going "Oh, yeah, got 10tb lying around." |
|
18:15
🔗
|
SketchCow |
The main critical thing is to make sure we have chosen collections that aren't wasteful. |
|
18:18
🔗
|
DFJustin |
I'm a little suspicious about some of the recent ones like wikipediadumps |
|
18:20
🔗
|
tpw_rules |
(why is that plural? i thought one dump contained the entire history of everything) |
|
18:21
🔗
|
DFJustin |
well there is more than one wikipedia (language) |
|
18:25
🔗
|
tpw_rules |
oh, true |
|
18:35
🔗
|
SketchCow |
I think wikipediadumps is on the edge. |
|
18:35
🔗
|
SketchCow |
On the other hand, our collection of dumps goes WAY back farther than anyone. |
|
18:35
🔗
|
SketchCow |
I did some of those, with lots of skeletons |
|
18:36
🔗
|
SketchCow |
Like Erik Moller pro-childporn arguments that were quietly expunged when he became Wikipedia org dude |
|
18:36
🔗
|
SketchCow |
Or Jimbo Wales getting into an argument with someone, and having a db admin remove the thing he said, and then going "I never said that." |
|
18:36
🔗
|
SketchCow |
And who knows what else, down there. |
|
18:36
🔗
|
SketchCow |
But year. |
|
18:36
🔗
|
SketchCow |
Yeah. |
|
18:36
🔗
|
SketchCow |
Maybe we need a nominations page. |
|
18:37
🔗
|
SketchCow |
Yes. |
|
18:37
🔗
|
SketchCow |
We do. |
|
18:37
🔗
|
SketchCow |
OK, one moment. |
|
18:37
🔗
|
|
Start has quit IRC (Read error: Connection reset by peer) |
|
18:46
🔗
|
SketchCow |
http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/nominations |
|
18:49
🔗
|
Beardface |
are the shars a set size? |
|
18:49
🔗
|
Beardface |
shards* |
|
18:49
🔗
|
SketchCow |
The shards are a set number of files/items |
|
18:50
🔗
|
SketchCow |
So imagine it's... 20 |
|
18:50
🔗
|
SketchCow |
20 1m files mean tiny shard |
|
18:50
🔗
|
SketchCow |
20 1g files means fatty shard |
|
18:50
🔗
|
SketchCow |
Once things are going into the upper reaches/echelons, we'll see cases where shards 100-1000 are all 1mb or some such chicanery |
|
18:51
🔗
|
Beardface |
ahh |
|
18:53
🔗
|
iabak-reg |
03registrar 05master a1cadb4 06other 10SHARD6/pubkeys registration of mmein301+i on SHARD6 |
|
18:53
🔗
|
SketchCow |
Like, 80% of the work going on now is to find and deal with use cases and bugs/contingencies they reveal. |
|
18:53
🔗
|
SketchCow |
10% is improving the UI and interaction |
|
18:54
🔗
|
SketchCow |
10% is filler, primarily melted hooves and horns |
|
19:00
🔗
|
iabak-reg |
03registrar 05master 95ae19f 06other 10SHARD6/pubkeys registration of nico+iabak on SHARD6 |
|
19:01
🔗
|
mariusz |
btw. does any one knows why git config annex.web-options=--limit-rate=200k returns "invalid key" error? |
|
19:01
🔗
|
hendi |
If I want to run iabak on multiple machines, should I copy the private key over for accounting and stuff, or use a new one on each machine? |
|
19:08
🔗
|
iabak-reg |
03registrar 05master b6d2ef7 06other 10SHARD6/pubkeys registration of cyberjacob+IA on SHARD6 |
|
19:08
🔗
|
closure |
mariusz: I think git config doesn't want the = there |
|
19:08
🔗
|
closure |
hendi: use a new one |
|
19:12
🔗
|
mariusz |
closure: that worked. thanks. probably would be good |
|
19:13
🔗
|
mariusz |
to change the README |
|
19:13
🔗
|
mariusz |
;) |
|
19:13
🔗
|
Senji |
Glad to see I'm not the only person using a foo+bar address :) |
|
19:14
🔗
|
iabak-reg |
03registrar 05master 1d18796 06other 10SHARD7/pubkeys registration of mail on SHARD7 |
|
19:15
🔗
|
tpw_rules |
error: Ref refs/heads/synced/git-annex is at 38179c6b1a70682556e88bf6d5c94187cdaabaac but expected ffeac6bf96c796c6117981d2ee64fc642edbaa01 |
|
19:16
🔗
|
tpw_rules |
i am still getting sync problems like that |
|
19:16
🔗
|
tpw_rules |
i don't have more than one iabak script running |
|
19:18
🔗
|
closure |
well, that can happen if someone else pushed a change at the same time. It should normally clear up the next time, unless you're unludky |
|
19:24
🔗
|
iabak-reg |
03registrar 05master f760a1f 06other 10SHARD7/pubkeys registration of cyberjacob+IA on SHARD7 |
|
19:28
🔗
|
|
Start has joined #internetarchive.bak |
|
19:30
🔗
|
|
Start has quit IRC (Client Quit) |
|
19:31
🔗
|
iabak-reg |
03registrar 05master c7d5ea9 06other 10SHARD6/pubkeys registration of eskild on SHARD6 |
|
19:48
🔗
|
|
beardicus has joined #internetarchive.bak |
|
19:50
🔗
|
iabak-reg |
03registrar 05master 40e077c 06other 10SHARD7/pubkeys registration of moritz.steiner on SHARD7 |
|
19:50
🔗
|
|
primus104 has joined #internetarchive.bak |
|
19:51
🔗
|
Zero_Dogg |
/win 20 |
|
19:52
🔗
|
Zero_Dogg |
bah |
|
19:53
🔗
|
sep332 |
where are the authorized_keys files again? I think I'm missing from shard1 |
|
19:54
🔗
|
|
atomotic has joined #internetarchive.bak |
|
19:57
🔗
|
closure |
.git/annex/id_rsa and id_rsa.pub |
|
19:58
🔗
|
|
CyberJaco has joined #internetarchive.bak |
|
19:58
🔗
|
CyberJaco |
Hi |
|
19:59
🔗
|
sep332 |
i have id_rsa but I'm getting Permission denied (publickey) when i sync |
|
19:59
🔗
|
sep332 |
hi CyberJaco |
|
19:59
🔗
|
iabak-reg |
03registrar 05master dd3be51 06other 10SHARD6/pubkeys registration of hannson on SHARD6 |
|
20:00
🔗
|
CyberJaco |
that's weird, why is the last letter of my name mising... |
|
20:00
🔗
|
closure |
sep332: sounds like the wrong key, we have separate sets of keys for each shard, so shard1 may not have the pubkey you're using for other shards |
|
20:00
🔗
|
sep332 |
looks like an 8-char limit? |
|
20:01
🔗
|
sep332 |
closure: can i register a new one? |
|
20:01
🔗
|
closure |
sep332: manually, yes.. |
|
20:01
🔗
|
closure |
./register-helper.pl "$SHARD" "$uuid" "$registrationemail" "$(cat id_rsa.pub)" |
|
20:02
🔗
|
closure |
full in the bits, that will give an url you can hit to register |
|
20:05
🔗
|
|
mariusz has quit IRC (Read error: Operation timed out) |
|
20:05
🔗
|
closure |
sep332: or, I can manually add it |
|
20:07
🔗
|
iabak-reg |
03registrar 05master f36f056 06other 10SHARD1/pubkeys registration of sean.palmer on SHARD1 |
|
20:08
🔗
|
iabak-reg |
03registrar 05master fd09178 06other 10SHARD6/pubkeys registration of brian on SHARD6 |
|
20:09
🔗
|
sep332 |
closure: is it the [annex] uuid or the [remote "origin"] annex-uuid? |
|
20:10
🔗
|
iabak-reg |
03registrar 05master 84567e0 06other 10SHARD6/pubkeys registration of steven.m.reed on SHARD6 |
|
20:11
🔗
|
SketchCow |
SHARD6 does the climb |
|
20:11
🔗
|
closure |
sep332: the annex.uuid |
|
20:17
🔗
|
sep332 |
closure: that's what i put in, same error |
|
20:17
🔗
|
sep332 |
i'm using the same key for all shards |
|
20:18
🔗
|
closure |
check perms of your id_rsa file |
|
20:19
🔗
|
sep332 |
-rw------- |
|
20:24
🔗
|
closure |
sep332: check if shard1's git config has remote.origin.annex-ssh-options set |
|
20:27
🔗
|
sep332 |
nope. i'll just copy it from another shard |
|
20:28
🔗
|
closure |
well, not the whole config, just that setting |
|
20:28
🔗
|
sep332 |
yeah |
|
20:29
🔗
|
sep332 |
alright it's working. thanks closure |
|
20:42
🔗
|
|
CyberJaco is now known as zz_CyberJ |
|
20:44
🔗
|
iabak-reg |
03registrar 05master 4de5003 06other 10SHARD6/pubkeys registration of mariusz on SHARD6 |
|
20:46
🔗
|
|
sankin has quit IRC (Leaving.) |
|
20:50
🔗
|
SketchCow |
awww yes here comes mariusz |
|
20:56
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
|
21:01
🔗
|
|
xhdr has joined #internetarchive.bak |
|
21:10
🔗
|
tpw_rules |
closure: i think the sync error may have killed the iabak-hourly process. it's been a couple hours since that error and it hasn't tried again |
|
21:21
🔗
|
closure |
indeed, that could happen |
|
21:22
🔗
|
|
laxity has joined #internetarchive.bak |
|
21:29
🔗
|
lhobas_ |
closure: noticed that the cleanup function in the iabak script doesn't work on OS X (more stupid Mac-only glitches I assume) - https://github.com/ArchiveTeam/IA.BAK/blob/a420ad/iabak-helper#L282 throws "No such file or directory" (pid does exist, statement should eval to true) |
|
21:29
🔗
|
lhobas_ |
any clue what could cause that? |
|
21:29
🔗
|
lhobas_ |
(#L285 in that file seems off to me, think the file is supposed to be rm'ed, not the pid-number right?) |
|
21:34
🔗
|
iabak-reg |
03registrar 05master e16b5ee 06other 10SHARD6/pubkeys registration of matt on SHARD6 |
|
22:12
🔗
|
db48x |
I should have gone ahead and fixed that last night when you mentioned it |
|
22:12
🔗
|
db48x |
couldn't sleep anyway |
|
22:15
🔗
|
tpw_rules |
do you not love me |
|
22:16
🔗
|
closure |
SketchCow on npr, eep |
|
22:23
🔗
|
iabak-reg |
03registrar 05master 93b954b 06other 10SHARD6/pubkeys registration of carl.moden on SHARD6 |
|
22:39
🔗
|
|
mariusz has joined #internetarchive.bak |
|
23:05
🔗
|
|
Atluxity has quit IRC (Ping timeout: 360 seconds) |
|
23:10
🔗
|
|
Atluxity has joined #internetarchive.bak |
|
23:12
🔗
|
iabak-reg |
03registrar 05master 240d6ac 06other 10SHARD6/pubkeys registration of paul.chambers on SHARD6 |
|
23:18
🔗
|
|
beardicus has quit IRC (Quit: Sleep.) |
|
23:19
🔗
|
|
Atluxity has quit IRC (Ping timeout: 360 seconds) |
|
23:23
🔗
|
|
Start has joined #internetarchive.bak |
|
23:26
🔗
|
|
Atluxity has joined #internetarchive.bak |
|
23:27
🔗
|
|
beardicus has joined #internetarchive.bak |
|
23:31
🔗
|
db48x |
closure: ah, an interesting clue |
|
23:32
🔗
|
db48x |
so where is the ffi function that calls CreateProcess? |
|
23:36
🔗
|
db48x |
I'm looking at http://hackage.haskell.org/package/process-1.2.3.0/docs/src/System-Process.html#createProcess, but I don't see where it actually calls the win32 api... |
|
23:37
🔗
|
|
Atluxity has quit IRC (Ping timeout: 360 seconds) |
|
23:38
🔗
|
|
Atluxity has joined #internetarchive.bak |
|
23:45
🔗
|
closure |
db48x: oh, I just nailed that problem |
|
23:45
🔗
|
|
zottelbey has quit IRC (Quit: Leaving) |
|
23:46
🔗
|
closure |
writing the PR for the library that needs changes now.. |
|
23:46
🔗
|
closure |
you were in the right place, but it has a side of C files :) |
|
23:49
🔗
|
|
Atluxity has quit IRC (Ping timeout: 360 seconds) |
|
23:52
🔗
|
iabak-reg |
03registrar 05master de4a487 06other 10SHARD6/pubkeys registration of iabak on SHARD6 |
|
23:53
🔗
|
db48x |
ah, good |
|
23:53
🔗
|
db48x |
is https://github.com/haskell/process/blob/master/System/Process/Internals.hs#L414 closer? |
|
23:54
🔗
|
db48x |
ah, https://github.com/haskell/process/blob/master/cbits/runProcess.c#L557 |
|
23:56
🔗
|
db48x |
presumably you're adding a way to add that flag where that's called here: https://github.com/haskell/process/blob/master/System/Process/Internals.hs#L452 |
|
23:57
🔗
|
closure |
that's the plan, but I'm actually watching austraian cooking show :P |
|
23:57
🔗
|
closure |
feel free to send patch to https://github.com/haskell/process/issues/32 |
|
23:58
🔗
|
|
Atluxity has joined #internetarchive.bak |