| Time |
Nickname |
Message |
|
00:04
π
|
|
signius has joined #archiveteam |
|
00:17
π
|
|
dashcloud has quit IRC (Read error: Connection reset by peer) |
|
00:17
π
|
|
dashcloud has joined #archiveteam |
|
00:28
π
|
|
BlueMaxim has joined #archiveteam |
|
00:56
π
|
|
Rotab has quit IRC (hub.se irc.du.se) |
|
01:27
π
|
|
Boppen has joined #archiveteam |
|
01:30
π
|
|
Boppen has quit IRC (hub.se irc.du.se) |
|
01:48
π
|
|
xtr-201 has joined #archiveteam |
|
02:02
π
|
|
dashcloud has quit IRC (Read error: Connection reset by peer) |
|
02:02
π
|
|
dashcloud has joined #archiveteam |
|
02:07
π
|
|
mistym has quit IRC (Remote host closed the connection) |
|
02:13
π
|
|
primus104 has quit IRC (Leaving.) |
|
02:21
π
|
|
mistym has joined #archiveteam |
|
02:28
π
|
|
BlueMaxim has quit IRC (Read error: Operation timed out) |
|
02:28
π
|
|
BlueMaxim has joined #archiveteam |
|
03:18
π
|
|
BlueMaxim has quit IRC (Read error: Connection reset by peer) |
|
03:19
π
|
|
garyrh has quit IRC (Remote host closed the connection) |
|
03:42
π
|
|
garyrh has joined #archiveteam |
|
04:11
π
|
|
BlueMaxim has joined #archiveteam |
|
04:30
π
|
|
VonGuard_ is now known as VonGuard |
|
05:09
π
|
|
antomatic has quit IRC (Read error: Connection reset by peer) |
|
05:09
π
|
|
lytv has quit IRC (Read error: Connection reset by peer) |
|
05:09
π
|
|
fresco___ has quit IRC (hub.dk efnet.port80.se) |
|
05:09
π
|
|
VonGuard has quit IRC (hub.dk efnet.port80.se) |
|
05:09
π
|
|
russss has quit IRC (hub.dk efnet.port80.se) |
|
05:09
π
|
|
deathy has quit IRC (hub.dk efnet.port80.se) |
|
05:09
π
|
|
danneh_ has quit IRC (hub.dk efnet.port80.se) |
|
05:09
π
|
|
LittUp has quit IRC (hub.dk efnet.port80.se) |
|
05:09
π
|
|
Muad-Dib has quit IRC (hub.dk efnet.port80.se) |
|
05:09
π
|
|
Rickster has quit IRC (hub.dk efnet.port80.se) |
|
05:09
π
|
|
lhobas has quit IRC (hub.dk efnet.port80.se) |
|
05:09
π
|
|
nox has quit IRC (Read error: Operation timed out) |
|
05:09
π
|
|
NovaKing_ has quit IRC (Read error: Operation timed out) |
|
05:09
π
|
|
yipdw has quit IRC (hub.dk irc.homelien.no) |
|
05:09
π
|
|
pikhq has quit IRC (hub.dk irc.homelien.no) |
|
05:09
π
|
|
altlabel has quit IRC (hub.dk irc.homelien.no) |
|
05:09
π
|
|
ionpulse has quit IRC (hub.dk irc.homelien.no) |
|
05:09
π
|
|
antomati_ has joined #archiveteam |
|
05:09
π
|
|
NovaKing_ has joined #archiveteam |
|
05:09
π
|
|
nox has joined #archiveteam |
|
05:11
π
|
|
lytv has joined #archiveteam |
|
05:17
π
|
|
antomati_ has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
Zebranky_ has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
Fusl has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
ryan__ has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
ruukasu has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
Deewiant has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
edsu_ has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
Kazzy has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
ex-parrot has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
Gfy has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
SketchCow has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
w0rp has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
Sellyme_ has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
jk[SVP] has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
Kniffy has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
Kenshin has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
Nemo_bis has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
yan has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
nico_32 has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
raylee has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
Atluxity has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
is- has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
nox has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
NovaKing_ has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
espes__ has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
aNthraXx has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
cadbury_ has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
underscor has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
Sue__ has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
gibigian1 has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
kanzure_ has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
lukeman has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
warthurto has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
Sk1d has quit IRC (hub.efnet.us hub.dk) |
|
05:17
π
|
|
Void_ has quit IRC (hub.efnet.us hub.dk) |
|
05:21
π
|
|
espes___ has joined #archiveteam |
|
05:48
π
|
|
dashcloud has quit IRC (Quit: No Ping reply in 210 seconds.) |
|
05:50
π
|
|
dashcloud has joined #archiveteam |
|
06:26
π
|
|
dashcloud has quit IRC (Read error: Connection reset by peer) |
|
06:27
π
|
|
dashcloud has joined #archiveteam |
|
07:15
π
|
|
cadbury_ has joined #archiveteam |
|
07:15
π
|
|
lhobas has joined #archiveteam |
|
07:15
π
|
|
Muad-Dib has joined #archiveteam |
|
07:15
π
|
|
Rickster has joined #archiveteam |
|
07:15
π
|
|
danneh_ has joined #archiveteam |
|
07:15
π
|
|
LittUp has joined #archiveteam |
|
07:15
π
|
|
deathy has joined #archiveteam |
|
07:15
π
|
|
russss has joined #archiveteam |
|
07:15
π
|
|
VonGuard has joined #archiveteam |
|
07:15
π
|
|
fresco___ has joined #archiveteam |
|
07:15
π
|
|
warthurto has joined #archiveteam |
|
07:15
π
|
|
lukeman has joined #archiveteam |
|
07:15
π
|
|
Sue__ has joined #archiveteam |
|
07:15
π
|
|
aNthraXx has joined #archiveteam |
|
07:15
π
|
|
Void_ has joined #archiveteam |
|
07:15
π
|
|
Rotab has joined #archiveteam |
|
07:15
π
|
|
underscor has joined #archiveteam |
|
07:15
π
|
|
ionpulse has joined #archiveteam |
|
07:15
π
|
|
altlabel has joined #archiveteam |
|
07:15
π
|
|
pikhq has joined #archiveteam |
|
07:15
π
|
|
yipdw has joined #archiveteam |
|
07:15
π
|
|
gibigiana has joined #archiveteam |
|
07:15
π
|
|
Sk1d has joined #archiveteam |
|
07:15
π
|
|
antomati_ has joined #archiveteam |
|
07:15
π
|
|
Nemo_bis has joined #archiveteam |
|
07:15
π
|
|
yan has joined #archiveteam |
|
07:15
π
|
|
nico_32 has joined #archiveteam |
|
07:15
π
|
|
Fusl has joined #archiveteam |
|
07:15
π
|
|
Zebranky_ has joined #archiveteam |
|
07:15
π
|
|
ryan__ has joined #archiveteam |
|
07:15
π
|
|
is- has joined #archiveteam |
|
07:15
π
|
|
ruukasu has joined #archiveteam |
|
07:15
π
|
|
Deewiant has joined #archiveteam |
|
07:15
π
|
|
raylee has joined #archiveteam |
|
07:15
π
|
|
edsu_ has joined #archiveteam |
|
07:15
π
|
|
Kazzy has joined #archiveteam |
|
07:15
π
|
|
ex-parrot has joined #archiveteam |
|
07:15
π
|
|
jk[SVP] has joined #archiveteam |
|
07:15
π
|
|
Sellyme_ has joined #archiveteam |
|
07:15
π
|
|
w0rp has joined #archiveteam |
|
07:15
π
|
|
SketchCow has joined #archiveteam |
|
07:15
π
|
|
Gfy has joined #archiveteam |
|
07:15
π
|
|
Kenshin has joined #archiveteam |
|
07:15
π
|
|
Kniffy has joined #archiveteam |
|
07:15
π
|
|
Atluxity has joined #archiveteam |
|
07:15
π
|
|
hub.se sets mode: +ooo raylee SketchCow Kenshin |
|
07:15
π
|
|
swebb sets mode: +o underscor |
|
07:15
π
|
|
swebb sets mode: +o SketchCow |
|
07:17
π
|
|
kanzure has joined #archiveteam |
|
07:59
π
|
|
Jonimus has quit IRC (Ping timeout: 370 seconds) |
|
08:06
π
|
|
mistym has quit IRC (Remote host closed the connection) |
|
08:10
π
|
|
Jonimus has joined #archiveteam |
|
09:03
π
|
|
schbirid has joined #archiveteam |
|
09:04
π
|
|
dashcloud has quit IRC (Read error: Connection reset by peer) |
|
09:04
π
|
|
dashcloud has joined #archiveteam |
|
09:17
π
|
|
Ymgve has joined #archiveteam |
|
09:19
π
|
|
primus104 has joined #archiveteam |
|
09:39
π
|
|
antomati_ is now known as antomatic |
|
10:10
π
|
|
primus104 has quit IRC (Leaving.) |
|
10:22
π
|
|
Sk1d has quit IRC (Ping timeout: 265 seconds) |
|
10:25
π
|
|
Sk1d has joined #archiveteam |
|
10:45
π
|
|
Sk2d has joined #archiveteam |
|
10:46
π
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
10:46
π
|
|
Sk2d is now known as Sk1d |
|
11:25
π
|
|
dashcloud has quit IRC (Quit: No Ping reply in 180 seconds.) |
|
11:26
π
|
|
dashcloud has joined #archiveteam |
|
11:31
π
|
Muad-Dib |
Alright, I want to grab a big-ass mirror of a niche art site that includes a lot of stuff that has been οΏ½deletedοΏ½ from the net earlier, itοΏ½s probably multiple TBs and seems to have limited bandwidth, shall I just put it in archivebot or do we grab this seperately? http://vj5pbopejlhcbz4n.onion.city/indexes |
|
11:32
π
|
Muad-Dib |
οΏ½deletedοΏ½ from the site |
|
11:32
π
|
Muad-Dib |
* |
|
11:35
π
|
|
Sk1d has quit IRC (Ping timeout: 265 seconds) |
|
11:35
π
|
Ctrl-S |
I want a copy of this too |
|
11:36
π
|
Muad-Dib |
there's a lot of furry porn in there, lol :D |
|
11:36
π
|
Muad-Dib |
long live the internet |
|
11:37
π
|
Muad-Dib |
freaky place I'd trade for no other |
|
11:37
π
|
Muad-Dib |
Ctrl-S, you have terabytes available ATM? |
|
11:37
π
|
|
Sk1d has joined #archiveteam |
|
11:37
π
|
Muad-Dib |
I have a feeling this archive this might well pass the 10 TB |
|
11:37
π
|
Muad-Dib |
mark |
|
11:38
π
|
Ctrl-S |
I'm on a capped connection though :( |
|
11:38
π
|
Ctrl-S |
could you host it for a year or so so i can afford to grab a copy? |
|
11:39
π
|
Muad-Dib |
looks like there's no archivebot pipeline with enough storage for grabbing it all at once either :C http://dashboard.at.ninjawedding.org/pipelines |
|
11:39
π
|
Muad-Dib |
lol |
|
11:39
π
|
Muad-Dib |
Ctrl-S, you might as well hire dedi hosting, lol |
|
11:39
π
|
Muad-Dib |
for one month, grab everything, and post it to IA |
|
11:39
π
|
Ctrl-S |
i'm serious about wanting this mirrored |
|
11:39
π
|
Muad-Dib |
me too |
|
11:39
π
|
Ctrl-S |
what do i have to do to get it done? |
|
11:40
π
|
Muad-Dib |
but its way too much for me to hold |
|
11:40
π
|
Ctrl-S |
where do i send the drive money |
|
11:40
π
|
Muad-Dib |
It's already kicking up controversy in the art site's community for hosting people's old and deleted stuff |
|
11:40
π
|
Muad-Dib |
I don't expect it to be up for long |
|
11:40
π
|
Ctrl-S |
because mailing HDDs is the only way i can get a copy of this |
|
11:41
π
|
Muad-Dib |
don't expect it to be up for long on clearnet anywa |
|
11:41
π
|
Muad-Dib |
y |
|
11:41
π
|
Ctrl-S |
>controversy on furaffinity |
|
11:41
π
|
Muad-Dib |
IKR |
|
11:41
π
|
Muad-Dib |
"OH NO, I POSTED MY STUFF TO THE PUBLIC INTERNET AND I CANT GET RID OF IT ANYMORE" |
|
11:42
π
|
Ctrl-S |
can we contact the admin? |
|
11:42
π
|
Ctrl-S |
of this mirror i mean |
|
11:42
π
|
Muad-Dib |
no one knows who's hosting this |
|
11:42
π
|
Muad-Dib |
but it might be site staff, since it includes so many "deleted" files |
|
11:44
π
|
Ctrl-S |
I would seriously pay the several hundred dollars for disk space for this |
|
11:44
π
|
Ctrl-S |
because I KNOW it's endangered |
|
11:45
π
|
Atluxity |
would be nice to not archive this via onion.city, but rather do it via tor? looks like a hidden service proxy to me |
|
11:46
π
|
Muad-Dib |
maybe I should just throw it in archivebot and see how far it gets |
|
11:46
π
|
arkiver |
10TB is nothing for archivebot |
|
11:46
π
|
Muad-Dib |
Atluxity: ideally, yes |
|
11:46
π
|
arkiver |
if we want this we can create a warrior project |
|
11:46
π
|
Muad-Dib |
arkiver: http://dashboard.at.ninjawedding.org/pipelines |
|
11:46
π
|
Muad-Dib |
a warrior project that grabs shit from tor? |
|
11:46
π
|
arkiver |
yeah, why not |
|
11:47
π
|
Atluxity |
arkiver: warrior project getting archiving a tor hidden service? sounds...interesting |
|
11:47
π
|
arkiver |
onion.city for now |
|
11:47
π
|
Muad-Dib |
won't that require extra dependencies on the warrior VM's? |
|
11:47
π
|
Ctrl-S |
We DO need to get around to backing up the tor hidden sites |
|
11:47
π
|
Atluxity |
correct |
|
11:47
π
|
arkiver |
I can create a project for this .onion.city site easily |
|
11:47
π
|
arkiver |
but 10TB is a lot |
|
11:47
π
|
Muad-Dib |
but IA might not be willing to host hidden services, with good reason |
|
11:47
π
|
Ctrl-S |
they hold an especially high degree of cultural relevance due to their often illicit nature |
|
11:47
π
|
arkiver |
not sure if IA is willing to take that all in |
|
11:47
π
|
Muad-Dib |
talk to the onion.city admins first ;) |
|
11:48
π
|
schbirid |
<Muad-Dib> but IA might not be willing to host hidden services, with good reason |
|
11:48
π
|
schbirid |
also the opposite |
|
11:48
π
|
schbirid |
they might be very willing, with good reason |
|
11:48
π
|
Muad-Dib |
I know |
|
11:48
π
|
Ctrl-S |
they don't have to provide open access, just hang onto the data |
|
11:48
π
|
Muad-Dib |
I think they'd probably be a bit... conflicted about it |
|
11:48
π
|
arkiver |
if SketchCow thinks IA is willing to take multiple TB's from http://vj5pbopejlhcbz4n.onion.city/indexes |
|
11:49
π
|
arkiver |
if that ^ I'll have a project running soon |
|
11:51
π
|
Ctrl-S |
I actually wrote a script to save things from FA, but i'm on a capped connection so i can't save everything |
|
11:53
π
|
Muad-Dib |
<arkiver> 10TB is nothing for archivebot |
|
11:53
π
|
Muad-Dib |
3tb max free diskspace isn't agreeing with you, ark http://dashboard.at.ninjawedding.org/pipelines |
|
11:53
π
|
arkiver |
what I meant is that a website of 10TB whould |
|
11:53
π
|
arkiver |
shouldn't be archived with arcivebot |
|
11:54
π
|
Muad-Dib |
oh |
|
11:54
π
|
Muad-Dib |
okay |
|
11:54
π
|
Muad-Dib |
misinterpretation :P |
|
11:54
π
|
arkiver |
yep, I wasn't clear |
|
11:54
π
|
Ctrl-S |
>That wonderous feel when you find a copy of something yo'd long thought deleted |
|
11:55
π
|
Muad-Dib |
<3 |
|
11:56
π
|
Ctrl-S |
whatever the case, this needs backing up right now, and i will do anything in my power to help you do so |
|
11:57
π
|
Ctrl-S |
I've seen too many artists go bezerk and delete everything to lose this |
|
11:57
π
|
arkiver |
do you have 10TB of free diskspace? |
|
11:57
π
|
arkiver |
if you do, we'll start |
|
11:57
π
|
Ctrl-S |
maybe, but 1TB/month cap |
|
11:57
π
|
Ctrl-S |
Being australian is suffering |
|
12:21
π
|
Muad-Dib |
;_;7 |
|
12:52
π
|
|
nox has joined #archiveteam |
|
13:04
π
|
|
BlueMaxim has quit IRC (Read error: Connection reset by peer) |
|
13:15
π
|
* |
ersi points and laughs |
|
13:26
π
|
Rotab |
;) |
|
13:54
π
|
|
primus104 has joined #archiveteam |
|
13:56
π
|
|
sankin has joined #archiveteam |
|
14:32
π
|
Muad-Dib |
https://www.youtube.com/watch?v=EWCLpaynj4Y fuck my country and its people ;_; |
|
14:32
π
|
Muad-Dib |
but at least we dont have bandwidth caps ;) |
|
14:40
π
|
midas |
rofl Muad-Dib |
|
14:41
π
|
Muad-Dib |
white trash, white trash everywhere ;_; |
|
14:41
π
|
midas |
aye |
|
14:41
π
|
Muad-Dib |
glorious YUROP |
|
14:42
π
|
|
aNthraXx has quit IRC (Read error: Operation timed out) |
|
14:43
π
|
|
aNthraXx has joined #archiveteam |
|
15:19
π
|
|
Start has quit IRC (Disconnected.) |
|
15:25
π
|
|
Sk2d has joined #archiveteam |
|
15:28
π
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
15:29
π
|
|
Sk1d has joined #archiveteam |
|
15:30
π
|
|
Sk2d has quit IRC (Ping timeout: 265 seconds) |
|
15:33
π
|
|
Froggypwn has quit IRC (Read error: Operation timed out) |
|
15:34
π
|
|
Froggypwn has joined #archiveteam |
|
15:35
π
|
arkiver |
midas: are you able to get the list of ftps back online? |
|
15:36
π
|
midas |
is it offline? |
|
15:37
π
|
arkiver |
yeah, 503 |
|
15:37
π
|
arkiver |
502* |
|
15:39
π
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
15:40
π
|
midas |
stupid pad crashed |
|
15:43
π
|
|
Sk1d has joined #archiveteam |
|
15:51
π
|
|
dashcloud has quit IRC (Read error: Connection reset by peer) |
|
15:51
π
|
|
Sk1d has quit IRC (Ping timeout: 265 seconds) |
|
15:53
π
|
|
Sk1d has joined #archiveteam |
|
15:56
π
|
|
dashcloud has joined #archiveteam |
|
15:58
π
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
16:00
π
|
|
mistym has joined #archiveteam |
|
16:01
π
|
|
mistym has quit IRC (Remote host closed the connection) |
|
16:04
π
|
|
Start has joined #archiveteam |
|
16:05
π
|
|
dashcloud has joined #archiveteam |
|
16:06
π
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
16:06
π
|
|
Sk2d has joined #archiveteam |
|
16:06
π
|
|
Sk2d is now known as Sk1d |
|
16:20
π
|
|
Sk1d has quit IRC (Ping timeout: 265 seconds) |
|
16:21
π
|
|
signius has quit IRC (Read error: Operation timed out) |
|
16:23
π
|
|
mistym has joined #archiveteam |
|
16:25
π
|
|
Sk1d has joined #archiveteam |
|
16:33
π
|
DFJustin |
fwiw archivebot uploads in 5gb intervals so you don't actually need 10tb of free space |
|
16:34
π
|
DFJustin |
tasks that run for months can be an issue though as machines need maintenance etc |
|
16:35
π
|
|
signius has joined #archiveteam |
|
16:37
π
|
DFJustin |
so if there's some way to feed in pieces of it one at a time (subdirectories are ideal) |
|
16:51
π
|
|
Start has quit IRC (Disconnected.) |
|
16:57
π
|
|
danneh_ has quit IRC (Ping timeout: 260 seconds) |
|
17:03
π
|
|
danneh_ has joined #archiveteam |
|
17:09
π
|
|
Nertsy has quit IRC (Read error: Operation timed out) |
|
17:17
π
|
|
mistym has quit IRC (Remote host closed the connection) |
|
17:31
π
|
|
mistym has joined #archiveteam |
|
17:35
π
|
|
sep332 has quit IRC (bye) |
|
17:37
π
|
|
sep332 has joined #archiveteam |
|
17:48
π
|
chfoo |
i can probably implement tor for archivebot sometime this week |
|
17:50
π
|
arkiver |
chfoo: I do think 10TB websites shouldn't be done with archivebot |
|
17:52
π
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
17:55
π
|
|
Sk1d has joined #archiveteam |
|
18:02
π
|
|
Start has joined #archiveteam |
|
18:02
π
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
18:04
π
|
|
Sk1d has joined #archiveteam |
|
18:09
π
|
|
Sk1d has quit IRC (Ping timeout: 265 seconds) |
|
18:09
π
|
chfoo |
https://about.gitlab.com/2015/03/03/gitlab-acquires-gitorious/ |
|
18:12
π
|
|
Sk1d has joined #archiveteam |
|
18:17
π
|
|
Sk2d has joined #archiveteam |
|
18:20
π
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
18:22
π
|
|
rolfb has joined #archiveteam |
|
18:22
π
|
|
Sk2d has quit IRC (Ping timeout: 265 seconds) |
|
18:23
π
|
|
Sk1d has joined #archiveteam |
|
18:24
π
|
rolfb |
Hi there. Gitorious has been acquired and gitorious.org will shut down at the end of May. Is there any way to preserve the data? |
|
18:30
π
|
|
Sk1d has quit IRC (Ping timeout: 265 seconds) |
|
18:31
π
|
chfoo |
arkiver: i'm not really fond of using tor in the warrior because it will involve setting up the latest tor and http proxy and it's likely that a manual script runner will break something. i'm also worried about needing to set up the warriors to use bridges in case the isp blocks tor |
|
18:31
π
|
chfoo |
but maybe someone with lots of bandwidth could set up a public tor proxy for archiveteam use |
|
18:33
π
|
|
Sk1d has joined #archiveteam |
|
18:39
π
|
chazchaz |
rolfb: Is there data that isn't already in the WayBackMachine? |
|
18:39
π
|
|
Sk2d has joined #archiveteam |
|
18:41
π
|
|
Start has quit IRC (Disconnected.) |
|
18:41
π
|
chazchaz |
As far as I can see, everything they have there other than the repo for the community edition source code is private/paid subscriptopn based. |
|
18:42
π
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
18:42
π
|
|
Sk2d is now known as Sk1d |
|
18:43
π
|
chfoo |
git clone everything |
|
18:45
π
|
chazchaz |
Wait, never mind, it appears they do host some repos |
|
18:49
π
|
chazchaz |
Apparently, GitLab took enough paying ustomers that Gitorious can't support its self while offering free service. |
|
18:50
π
|
|
Sk1d has quit IRC (Ping timeout: 265 seconds) |
|
18:51
π
|
|
abartov has joined #archiveteam |
|
19:00
π
|
|
Sk2d has joined #archiveteam |
|
19:01
π
|
|
kyan_ has joined #archiveteam |
|
19:03
π
|
|
kyan has quit IRC (Read error: Operation timed out) |
|
19:05
π
|
|
Sk1d- has joined #archiveteam |
|
19:06
π
|
|
Sk2d has quit IRC (Ping timeout: 265 seconds) |
|
19:09
π
|
|
Sk2d has joined #archiveteam |
|
19:09
π
|
|
Sk2d is now known as Sk1d |
|
19:11
π
|
|
Sk1d- has quit IRC (Read error: Operation timed out) |
|
19:11
π
|
fenn |
"We don't want to move people's code to another organization without their permission." yes, their open-source, public code |
|
19:14
π
|
|
Sk1d has quit IRC (Ping timeout: 265 seconds) |
|
19:14
π
|
Muad-Dib |
lol |
|
19:14
π
|
Muad-Dib |
sad |
|
19:18
π
|
|
Sk1d has joined #archiveteam |
|
19:21
π
|
|
sankin has quit IRC (Leaving.) |
|
19:22
π
|
|
Sk1d has quit IRC (Ping timeout: 265 seconds) |
|
19:25
π
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
19:31
π
|
|
Sk1d has joined #archiveteam |
|
19:32
π
|
|
dashcloud has joined #archiveteam |
|
19:33
π
|
rolfb |
chazchaz: sorry for not replying, i don't know what the waybackmachine has, but surely it would be more interesting to have the git repositories, and all the code on .org is available for download, over 100k repositories |
|
19:35
π
|
yipdw |
Muad-Dib: maximum diskspace has no effect on maximum job size |
|
19:35
π
|
yipdw |
the main problem with 10 TB is justifying shoving 10 TB into IA |
|
19:36
π
|
yipdw |
also running up someone's bandwidth bill if empathy is something you believe in |
|
19:43
π
|
|
BlueMaxim has joined #archiveteam |
|
19:48
π
|
sep332 |
I don't think it's bandwidth that's the problem. It takes more than 30 seconds to start getting data for some of those links |
|
19:57
π
|
yipdw |
I was referring also to the node operator's bill |
|
19:57
π
|
yipdw |
OVH doesn't seem to care, DO seems to eventually |
|
19:57
π
|
yipdw |
in any case a 10 TB job is really just a dick move at present time |
|
20:01
π
|
|
Start has joined #archiveteam |
|
20:03
π
|
ersi |
rolfb: waybackmachine = http://web.archive.org/ |
|
20:07
π
|
chfoo |
rolfb: are you the rolf the gitlab news is talking about? |
|
20:17
π
|
chfoo |
a database and data dump of everything straight from the source would be the most ideal |
|
20:23
π
|
chfoo |
second option would be a backdoor for archiveteam |
|
20:24
π
|
|
aschmitz has quit IRC (Read error: Operation timed out) |
|
20:28
π
|
|
Start has quit IRC (Disconnected.) |
|
20:31
π
|
|
aschmitz has joined #archiveteam |
|
20:49
π
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
20:55
π
|
|
dashcloud has joined #archiveteam |
|
21:13
π
|
|
Ymgve__ has joined #archiveteam |
|
21:18
π
|
|
Nertsy has joined #archiveteam |
|
21:18
π
|
|
Ymgve has quit IRC (Ping timeout: 506 seconds) |
|
21:19
π
|
|
cbb has joined #archiveteam |
|
21:20
π
|
|
Ymgve has joined #archiveteam |
|
21:22
π
|
|
Ymgve__ has quit IRC (Ping timeout: 506 seconds) |
|
21:26
π
|
|
Ymgve has quit IRC (Remote host closed the connection) |
|
21:26
π
|
|
Ymgve has joined #archiveteam |
|
21:28
π
|
|
Start has joined #archiveteam |
|
21:29
π
|
|
Start has quit IRC (Read error: Connection reset by peer) |
|
21:46
π
|
|
Start has joined #archiveteam |
|
21:58
π
|
Ctrl-S |
if it's diskspace that's the problem i can donate a few hundred bucks for drives for that FA dump |
|
22:03
π
|
|
Sk2d has joined #archiveteam |
|
22:04
π
|
|
mistym has quit IRC (Remote host closed the connection) |
|
22:04
π
|
rolfb |
chfoo: i am |
|
22:04
π
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
22:04
π
|
|
Sk2d is now known as Sk1d |
|
22:06
π
|
|
mistym has joined #archiveteam |
|
22:08
π
|
|
SN4T14_ has quit IRC (Read error: Connection reset by peer) |
|
22:09
π
|
|
Sk1d has quit IRC (Ping timeout: 265 seconds) |
|
22:09
π
|
|
SN4T14 has joined #archiveteam |
|
22:11
π
|
rolfb |
Ctrl-S: was that directed at me? |
|
22:11
π
|
Ctrl-S |
no |
|
22:11
π
|
|
Sk1d has joined #archiveteam |
|
22:12
π
|
rolfb |
ok :) |
|
22:12
π
|
Ctrl-S |
I odn't think so |
|
22:12
π
|
rolfb |
Ctrl-S: what was it about? |
|
22:12
π
|
Ctrl-S |
art hosting site backup someone's made with pretty much all the stuff that was deleted from the site included |
|
22:13
π
|
Ctrl-S |
~10 TB was estimated |
|
22:14
π
|
chfoo |
rolfb: is possible to just upload the repos directly to archive.org? |
|
22:14
π
|
rolfb |
chfoo: we have root, so I guess we can do whatever we want? we don't have much in terms of space to create images though |
|
22:15
π
|
xmc |
how much temporary space would you need? |
|
22:15
π
|
rolfb |
xmc: we have 4.5 TB of data |
|
22:16
π
|
xmc |
oh, so a reasonable amount |
|
22:16
π
|
rolfb |
always reasonable ;-) |
|
22:16
π
|
xmc |
:) |
|
22:16
π
|
Smiley |
I have a slooooooooow 2Tb |
|
22:16
π
|
xmc |
you could probably fire up an amazon instance with a bunch of storage for a few dozen bucks |
|
22:16
π
|
xmc |
and stream it to that for packaging |
|
22:17
π
|
|
schbirid has quit IRC (Leaving) |
|
22:19
π
|
Smiley |
the b/w in out tho?? |
|
22:19
π
|
|
Start has quit IRC (Disconnected.) |
|
22:20
π
|
rolfb |
Smiley: bandwidth is adjustable |
|
22:20
π
|
rolfb |
atleast on our side |
|
22:21
π
|
Smiley |
Nod |
|
22:21
π
|
Smiley |
but costs to export from amazon can be wild... |
|
22:21
π
|
rolfb |
we could possibly send physical disks |
|
22:21
π
|
Smiley |
oooooooooo |
|
22:21
π
|
rolfb |
but how would it be made available after? |
|
22:21
π
|
Smiley |
SketchCow could maybe accept physical disks |
|
22:21
π
|
Smiley |
well, if you have disks I'd think IA would host it |
|
22:21
π
|
Ctrl-S |
send disks to IA, IA uploads from the disks |
|
22:21
π
|
Smiley |
it's just the fact their storage costs like $1000/Tb |
|
22:21
π
|
Ctrl-S |
that much? |
|
22:22
π
|
Smiley |
yah due to duplication etc etc |
|
22:22
π
|
xmc |
ten cents a gig a month |
|
22:22
π
|
* |
Smiley can't remember exactly |
|
22:22
π
|
xmc |
IA or S3? |
|
22:22
π
|
xmc |
thousand gigs is a hundred bucks a month |
|
22:22
π
|
xmc |
ish |
|
22:23
π
|
xmc |
killer is transit from AWS, they estimate about 500 bux to get 5T out of AWS |
|
22:23
π
|
sep332 |
IA is $2k/TB. not per year, that's forever. |
|
22:23
π
|
DFJustin |
for ia you have to keep in mind it's amortized out to infinity because you have to replace drives every couple years |
|
22:23
π
|
xmc |
aye |
|
22:23
π
|
|
Panasonic has quit IRC (Ping timeout: 370 seconds) |
|
22:23
π
|
chfoo |
#archiveteam-bs |
|
22:24
π
|
rolfb |
sep332: meaning that if we send disks to IA, we need to pay them $9k to preserve the data? |
|
22:24
π
|
Smiley |
no |
|
22:24
π
|
Ctrl-S |
no, they have to pay that |
|
22:24
π
|
Smiley |
if you send htem disks, they'd be happy |
|
22:24
π
|
rolfb |
ok, ok |
|
22:24
π
|
Smiley |
if we want them to store the data for us, we might need to look at fundraising... |
|
22:24
π
|
Ctrl-S |
you only need the disks, if they can't find the space i presume they'd just keep the data somewhere less expensive |
|
22:25
π
|
Ctrl-S |
like in a cupboard |
|
22:25
π
|
rolfb |
but ... how would the git repositories be made available? |
|
22:25
π
|
xmc |
best practice for git repos is to export git bundles |
|
22:25
π
|
Ctrl-S |
zip of each repo, infopage as html as well? |
|
22:25
π
|
DFJustin |
IA has been very generous about doing pretty much anything we send them for free, the dollar figures are just to keep things in perspective |
|
22:25
π
|
xmc |
then an IA item would consist of a git bundle and all the other stuff from the repo |
|
22:25
π
|
xmc |
rolfb: what services exactly do you have for each repo? |
|
22:26
π
|
xmc |
i mean, what stuff do you store |
|
22:26
π
|
rolfb |
not much aside from the repository |
|
22:26
π
|
xmc |
so not a wiki/bugtracker/filedump like github does |
|
22:27
π
|
rolfb |
there's a wiki |
|
22:27
π
|
rolfb |
but that's also a repository |
|
22:27
π
|
xmc |
great |
|
22:27
π
|
xmc |
so if i were doing this |
|
22:27
π
|
xmc |
i would create one IA item per repo, containing two git bundles, one each of the source code and of the wiki |
|
22:27
π
|
xmc |
git bundles are, conveniently, bzip'd |
|
22:28
π
|
xmc |
but i'm sure you already know that :) |
|
22:28
π
|
rolfb |
xmc, just to complicate things ... we have repositories by project |
|
22:28
π
|
xmc |
project? |
|
22:28
π
|
rolfb |
example https://gitorious.org/gitorious/ |
|
22:28
π
|
xmc |
ahh |
|
22:28
π
|
rolfb |
but the project name could be metadata for a repo |
|
22:29
π
|
xmc |
right |
|
22:29
π
|
|
BlueMaxim has quit IRC (Ping timeout: 370 seconds) |
|
22:29
π
|
xmc |
i'd say put e.g. https://gitorious.org/gitorious/libdolt/ into http://archive.org/details/gitoriousexport_gitorious_libdolt |
|
22:30
π
|
xmc |
so the item names you're creating would be gitoriousexport_$(project)_$(repo) |
|
22:30
π
|
xmc |
and then you'd add various metadata fields to the item as well |
|
22:30
π
|
xmc |
how's this sound? |
|
22:30
π
|
rolfb |
sounds good |
|
22:30
π
|
xmc |
cool :) |
|
22:31
π
|
xmc |
you can use almost any characters in IA item names, but it's best practice to restrict to [-_A-Za-z0-9] |
|
22:31
π
|
xmc |
and . |
|
22:31
π
|
rolfb |
i'm pretty sure we have similar restrictions ... as names are used as urls |
|
22:31
π
|
xmc |
yeah |
|
22:31
π
|
xmc |
i've not heard of any characters except / breaking things ... but *shrug* |
|
22:32
π
|
rolfb |
but, how do we create an IA bundle? |
|
22:32
π
|
xmc |
ia bundle? |
|
22:32
π
|
rolfb |
item |
|
22:32
π
|
DFJustin |
https://pypi.python.org/pypi/internetarchive |
|
22:32
π
|
xmc |
there's a python toolo .. yes |
|
22:32
π
|
rolfb |
thanks |
|
22:33
π
|
xmc |
if you have all the items have a shared name prefix, or an identical metadata field, someone at IA can put them into a special collection |
|
22:34
π
|
rolfb |
is there a problem uploading 122k bundles? or should we rather send disks? |
|
22:34
π
|
rolfb |
ia items* |
|
22:35
π
|
xmc |
122,000 items / 4.5T? should be fine, i guess? especially if spread out over a month or so |
|
22:36
π
|
rolfb |
yup, something like that |
|
22:36
π
|
xmc |
the script that processes uploads will hold your upload until it's allocated space, which usually takes a few tens of seconds |
|
22:36
π
|
xmc |
so you might want to look into mild parallelism |
|
22:36
π
|
rolfb |
is this channel logged somewhere? |
|
22:36
π
|
xmc |
yes |
|
22:36
π
|
xmc |
also, i'm not an IA person |
|
22:36
π
|
DFJustin |
http://badcheese.com/~steve/atlogs/?chan=archiveteam |
|
22:36
π
|
xmc |
just a satisfied customer |
|
22:36
π
|
Ctrl-S |
i can give logs if you need them |
|
22:37
π
|
xmc |
rolfb: thanks for being a cool, forward-thinking person <3 |
|
22:37
π
|
rolfb |
my client has been logging so i'm all good for relaying information to the experts in my team |
|
22:38
π
|
xmc |
sweet |
|
22:38
π
|
rolfb |
xmc: since you are not an IA person, who do I verify that I can do this with? |
|
22:38
π
|
xmc |
SketchCow is an IA employee |
|
22:38
π
|
xmc |
i'd expect him to be in irc within the next few hours |
|
22:38
π
|
rolfb |
it's already past my bedtime |
|
22:38
π
|
rolfb |
<- norwegian |
|
22:39
π
|
xmc |
ahhh, yes |
|
22:39
π
|
rolfb |
xmc: also, thanks for the kind words |
|
22:39
π
|
xmc |
i know a finn elsewhere on efnet who went to bed an hour ago |
|
22:39
π
|
rolfb |
trying to make the best of a bad situation |
|
22:39
π
|
xmc |
you're a good sight better than most people in your situation |
|
22:39
π
|
Ctrl-S |
^this |
|
22:40
π
|
rolfb |
thanks, i'm just glad there is an alternative like IA |
|
22:40
π
|
rolfb |
xmc: will you be staying around till SketchCow arrives? |
|
22:40
π
|
DFJustin |
jscott@archive.org is his email |
|
22:40
π
|
xmc |
i'll be in and out. i'm working, and in a few hours i'll be going to beer |
|
22:40
π
|
rolfb |
ok, great. is it ok that I email him directly then? |
|
22:41
π
|
xmc |
but i'm in irc most of my waking life |
|
22:41
π
|
xmc |
yeah, go for it |
|
22:41
π
|
rolfb |
ok, any names I can use as referrals for getting in touch? |
|
22:41
π
|
rolfb |
or just use nicknames? |
|
22:41
π
|
xmc |
irc names is good |
|
22:42
π
|
DFJustin |
saying #archiveteam is probably good enough |
|
22:42
π
|
xmc |
"some people with @ before their name" |
|
22:42
π
|
xmc |
:P |
|
22:42
π
|
sep332 |
"i'm trying to rescue my shit" |
|
22:43
π
|
sep332 |
http://archiveteam.org/images/e/e6/Archiveteam.jpg |
|
22:44
π
|
rolfb |
:) |
|
22:48
π
|
|
Panasonic has joined #archiveteam |
|
22:52
π
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
22:54
π
|
|
BlueMaxim has joined #archiveteam |
|
22:59
π
|
|
dashcloud has joined #archiveteam |
|
22:59
π
|
rolfb |
email sent |
|
22:59
π
|
rolfb |
thanks again everyone |
|
23:01
π
|
|
mistym has quit IRC (Remote host closed the connection) |
|
23:12
π
|
|
rolfb has quit IRC (Linkinus - http://linkinus.com) |
|
23:20
π
|
|
mistym has joined #archiveteam |
|
23:21
π
|
|
Start has joined #archiveteam |
|
23:22
π
|
|
Panasonic has quit IRC (Ping timeout: 606 seconds) |