| Time |
Nickname |
Message |
|
00:29
🔗
|
|
primus104 has quit IRC (Leaving.) |
|
01:19
🔗
|
|
lbft has joined #archiveteam-bs |
|
01:23
🔗
|
|
Kazzy has quit IRC (Read error: Operation timed out) |
|
01:33
🔗
|
|
Kazzy has joined #archiveteam-bs |
|
01:40
🔗
|
|
Kazzy has quit IRC (Ping timeout: 265 seconds) |
|
01:40
🔗
|
|
twrist has quit IRC (And now, for my next magic trick..) |
|
01:44
🔗
|
|
Kazzy has joined #archiveteam-bs |
|
01:48
🔗
|
|
mistym has quit IRC (Remote host closed the connection) |
|
01:49
🔗
|
|
Kazzy has quit IRC (Ping timeout: 272 seconds) |
|
01:51
🔗
|
|
Kazzy has joined #archiveteam-bs |
|
01:57
🔗
|
|
Kazzy has quit IRC (Ping timeout: 365 seconds) |
|
02:02
🔗
|
|
Kazzy has joined #archiveteam-bs |
|
02:05
🔗
|
|
mistym has joined #archiveteam-bs |
|
02:08
🔗
|
|
DopefishJ has joined #archiveteam-bs |
|
02:08
🔗
|
|
swebb sets mode: +o DopefishJ |
|
02:10
🔗
|
|
dashcloud has quit IRC (Ping timeout: 265 seconds) |
|
02:10
🔗
|
|
DFJustin has quit IRC (Ping timeout: 265 seconds) |
|
02:10
🔗
|
|
Sellyme_ has quit IRC (Ping timeout: 265 seconds) |
|
02:10
🔗
|
|
Sellyme has joined #archiveteam-bs |
|
02:13
🔗
|
|
dashcloud has joined #archiveteam-bs |
|
02:14
🔗
|
|
Kazzy has quit IRC (Ping timeout: 272 seconds) |
|
02:17
🔗
|
|
amerrykan has quit IRC (Quit: Quitting) |
|
02:31
🔗
|
|
DopefishJ has quit IRC (Quit: IMHOSTFU) |
|
02:31
🔗
|
|
DFJustin has joined #archiveteam-bs |
|
02:31
🔗
|
|
swebb sets mode: +o DFJustin |
|
02:45
🔗
|
|
amerrykan has joined #archiveteam-bs |
|
03:08
🔗
|
|
schbirid2 has quit IRC (Read error: Operation timed out) |
|
03:09
🔗
|
|
schbirid2 has joined #archiveteam-bs |
|
03:40
🔗
|
|
ex-parrot has joined #archiveteam-bs |
|
04:30
🔗
|
|
Kazzy has joined #archiveteam-bs |
|
04:40
🔗
|
|
ex-parrot has quit IRC (Leaving.) |
|
04:51
🔗
|
|
aaaaaaaaa has quit IRC (Leaving) |
|
05:27
🔗
|
|
beardicus has quit IRC (Read error: Operation timed out) |
|
05:29
🔗
|
|
beardicus has joined #archiveteam-bs |
|
06:03
🔗
|
|
ex-parrot has joined #archiveteam-bs |
|
06:50
🔗
|
Kazzy |
wow, bumpy night over at ramnode. I rejoined 5 times overnight? |
|
07:04
🔗
|
|
Famicoman has quit IRC (Ping timeout: 369 seconds) |
|
07:08
🔗
|
|
robink has quit IRC (Ping timeout: 492 seconds) |
|
07:49
🔗
|
|
primus104 has joined #archiveteam-bs |
|
08:04
🔗
|
|
primus104 has quit IRC (Leaving.) |
|
08:19
🔗
|
|
robink has joined #archiveteam-bs |
|
08:36
🔗
|
midas |
bumpy night at efnet |
|
09:11
🔗
|
joepie91 |
^ |
|
09:11
🔗
|
joepie91 |
isn't it always :) |
|
09:16
🔗
|
midas |
since xs4all delinked it is |
|
09:28
🔗
|
godane |
hey everyone |
|
09:28
🔗
|
joepie91 |
lol |
|
09:28
🔗
|
godane |
i'm up at weird hours today |
|
09:29
🔗
|
godane |
but i have been getting up at 5 or 6 pm then going to bed at 8 am the last 2 days |
|
09:29
🔗
|
godane |
anyways 1972 pdfs of times news is uploaded |
|
09:32
🔗
|
midas |
meh |
|
09:32
🔗
|
midas |
uploading 2TB tar file via torrents > torrent file is over 150MB and too large to load :p |
|
09:33
🔗
|
|
mistym has quit IRC (Remote host closed the connection) |
|
09:34
🔗
|
godane |
try bigger pieces? |
|
09:34
🔗
|
godane |
like up it to 64 or 128mb chucks |
|
09:34
🔗
|
midas |
hm |
|
09:35
🔗
|
godane |
that would save on file size of torrent |
|
09:35
🔗
|
midas |
yeah |
|
09:36
🔗
|
godane |
may become a pain for people with slow connections but this is a 2tb tar file |
|
09:37
🔗
|
midas |
well it's just a upload to IA, so care for people with slow connections :p |
|
09:37
🔗
|
godane |
if the torrent is being uploaded to IA then they should be able to get it |
|
09:38
🔗
|
midas |
current issue is that rtorrent wont even load it on my side because it's huge :p |
|
09:39
🔗
|
godane |
ok |
|
09:40
🔗
|
midas |
changed it to the max size of mktorrent, lets see what it does |
|
09:47
🔗
|
joepie91 |
midas: gimme torrent file |
|
09:47
🔗
|
joepie91 |
I want to see whether qbittorrent can handle it |
|
09:47
🔗
|
joepie91 |
:P |
|
09:48
🔗
|
joepie91 |
aside; there's a torrent client SPECIFICALLY for seeding large (amounts of) torrent files |
|
09:48
🔗
|
joepie91 |
another aside: are you sure you want to upload a single 2TB TAR to IA :P |
|
10:02
🔗
|
midas |
yep :p |
|
10:03
🔗
|
midas |
already deleted torrent, creating new one |
|
10:22
🔗
|
arkiver |
<midas>uploading 2TB tar file via torrents > torrent file is over 150MB and too large to load :p |
|
10:22
🔗
|
arkiver |
remember to set the size when you upload |
|
10:33
🔗
|
midas |
arkiver: what do you mean? this is too big to load in rtorrent |
|
10:35
🔗
|
arkiver |
Oh, I didn't knew a torrent could be too big |
|
10:36
🔗
|
joepie91 |
oh hum, that's a good point |
|
10:36
🔗
|
joepie91 |
I hope IA can deal with this... |
|
10:36
🔗
|
joepie91 |
as in, grab the sizehint from the torrent.. |
|
10:37
🔗
|
arkiver |
we'll see |
|
10:43
🔗
|
midas |
in theory it should |
|
10:44
🔗
|
arkiver |
midas: do you already have it uploaded? the torrent? |
|
10:44
🔗
|
midas |
not yet, still hashing |
|
10:44
🔗
|
arkiver |
ah, ok |
|
10:44
🔗
|
midas |
Hashed 6444 of 7468 pieces. |
|
10:45
🔗
|
arkiver |
I'd like to see the deriving progress of the torrent in IA, so if you have it uploaded, please share the identifier with us :) |
|
10:45
🔗
|
arkiver |
just be sure to provide a size hint while uploading the torrent |
|
10:46
🔗
|
midas |
sure :) |
|
11:02
🔗
|
midas |
https://catalogd.archive.org/log/353604251 |
|
11:05
🔗
|
|
Bobby_ has joined #archiveteam-bs |
|
11:19
🔗
|
midas |
now we wait, alot |
|
11:22
🔗
|
midas |
probably first need to wait until archive.org actually connects to dht/tracker before it does anything :p |
|
11:23
🔗
|
arkiver |
looks good, seems to be working |
|
11:23
🔗
|
midas |
not seeing that much outbound traffic yet |
|
11:24
🔗
|
arkiver |
I'm always statrting the torrent when the torrent is already started by IA |
|
11:24
🔗
|
arkiver |
I don't know a lot about torrents. But you might want to try refreshing the trackers or stopping and starting the torrent |
|
11:24
🔗
|
|
brayden has quit IRC (Read error: Operation timed out) |
|
11:25
🔗
|
midas |
ill just wait |
|
11:25
🔗
|
arkiver |
ok |
|
11:25
🔗
|
arkiver |
how long did it take you to download that ftp? |
|
11:28
🔗
|
godane |
btw i'm finally uploading more of theblaze.com site |
|
11:29
🔗
|
godane |
i'm doing it by month cause trying to grab everything in wp-content path causes it to crash if i try to do the full year of 2011 |
|
11:36
🔗
|
midas |
arkiver: couple of weeks |
|
11:37
🔗
|
midas |
in total my ftp dump is now 22TB |
|
11:59
🔗
|
midas |
verification of data, waiting |
|
12:24
🔗
|
joepie91 |
FYI: Groupon is attempting to steamroll GNOME's trademarks by registering their own and releasing their own hardware and "operating system" named GNOME: http://gnome.org/groupon |
|
12:24
🔗
|
|
primus104 has joined #archiveteam-bs |
|
12:28
🔗
|
midas |
joepie91: added to archivebot for savekeeping |
|
12:28
🔗
|
joepie91 |
midas: yes, aware |
|
12:28
🔗
|
joepie91 |
that's where I originally heard of it |
|
12:28
🔗
|
joepie91 |
before freenode /notice'd it out |
|
12:28
🔗
|
joepie91 |
:) |
|
12:29
🔗
|
midas |
lol |
|
12:29
🔗
|
joepie91 |
I get a disturbing amount of my news from #archivebot |
|
12:29
🔗
|
joepie91 |
lol |
|
12:29
🔗
|
joepie91 |
it's pretty much my "shit ACTUALLY worth reading" newsfeed |
|
12:29
🔗
|
midas |
same |
|
12:29
🔗
|
midas |
via twitter and the channel, it's amazing |
|
12:31
🔗
|
joepie91 |
hehe, indeed |
|
12:31
🔗
|
arkiver |
bankruptcies ^ :) |
|
12:31
🔗
|
joepie91 |
midas: it's kinda funny when people ask me "what news sites do you read? because <insert mainstream site> kinda sucks..." |
|
12:32
🔗
|
joepie91 |
and I just go "uh... well... none, really... I just pick up stuff here and there..." |
|
12:32
🔗
|
midas |
err. well... i read https://twitter.com/archivebot |
|
12:36
🔗
|
arkiver |
midas: I see it didn't start yet |
|
12:36
🔗
|
arkiver |
maybe it's creathing the 1.8 TB file before starting to download? |
|
12:36
🔗
|
midas |
still loading data in transmission on my side it seems |
|
12:36
🔗
|
arkiver |
oh ok |
|
12:36
🔗
|
midas |
519.9 GB of 2.00 TB (25.9%) |
|
12:37
🔗
|
midas |
the issue with a atombox, single core work is sorta hard for it |
|
12:37
🔗
|
arkiver |
haha, that will take some time |
|
12:38
🔗
|
midas |
yep |
|
12:39
🔗
|
midas |
oh well, when it starts it should be fine |
|
12:39
🔗
|
arkiver |
I'm looking forward to seeing if it will actually get in, the full 2 TB |
|
12:40
🔗
|
midas |
same |
|
12:50
🔗
|
joepie91 |
midas: any particular reason you're using transmission? |
|
12:51
🔗
|
midas |
rtorrent didnt want to load it |
|
12:51
🔗
|
midas |
and headless box |
|
12:52
🔗
|
joepie91 |
midas: hold on |
|
12:52
🔗
|
* |
midas grabs table |
|
12:52
🔗
|
joepie91 |
lol |
|
12:52
🔗
|
joepie91 |
midas: there's a torrent client *specifically* for seeding many or large torrents |
|
12:52
🔗
|
joepie91 |
it does just that and nothing else |
|
12:52
🔗
|
joepie91 |
trying to find it for you |
|
12:52
🔗
|
midas |
ok |
|
12:52
🔗
|
midas |
still holding on to table, just to be sure |
|
12:54
🔗
|
joepie91 |
midas: http://www.pps.univ-paris-diderot.fr/~jch/software/hekate/ |
|
12:54
🔗
|
joepie91 |
:) |
|
12:54
🔗
|
joepie91 |
it's headless |
|
12:54
🔗
|
joepie91 |
and as I understand it, it's extremely low-overhead |
|
12:54
🔗
|
joepie91 |
so it should be perfect for this kind of thing |
|
12:55
🔗
|
joepie91 |
assuming your .torrent file has already been created, of course |
|
12:55
🔗
|
midas |
lol yep |
|
12:55
🔗
|
joepie91 |
okay :P |
|
12:55
🔗
|
joepie91 |
then it should do fine |
|
12:55
🔗
|
joepie91 |
protip: don't shut down transmission yet |
|
12:55
🔗
|
joepie91 |
both are single-core afaik so you can just run them both and see whether hekate is any faster |
|
12:55
🔗
|
midas |
lol nope ill leave her running :p |
|
12:55
🔗
|
joepie91 |
if it's not, then at least you won't lose your 500 gigs of work |
|
12:55
🔗
|
joepie91 |
lol |
|
12:55
🔗
|
midas |
650 now, amazing isnt it :p |
|
12:56
🔗
|
joepie91 |
lol |
|
12:56
🔗
|
joepie91 |
but yeah, try hekate :D |
|
12:56
🔗
|
midas |
doing that now |
|
12:56
🔗
|
midas |
BOOO! |
|
12:56
🔗
|
midas |
fatal: remote error: access denied or repository not exported: /hekate |
|
12:57
🔗
|
midas |
oh my |
|
12:57
🔗
|
midas |
the first link is just dead |
|
12:57
🔗
|
midas |
maybe we should just grab it in case of fire |
|
12:57
🔗
|
joepie91 |
heh |
|
12:58
🔗
|
joepie91 |
"in case of fire, please !a" |
|
12:58
🔗
|
midas |
yep :p |
|
12:58
🔗
|
joepie91 |
holy shit looks like my IA processing code is working so far |
|
12:58
🔗
|
|
Bobby__ has joined #archiveteam-bs |
|
12:58
🔗
|
joepie91 |
node.js streams are amazing :D |
|
12:58
🔗
|
joepie91 |
... speaking of the deviil |
|
12:58
🔗
|
midas |
screenshots yet? :p |
|
12:58
🔗
|
joepie91 |
ohai Bobby_ |
|
12:58
🔗
|
joepie91 |
er |
|
12:58
🔗
|
joepie91 |
Bobby__ |
|
12:59
🔗
|
joepie91 |
midas: screenshots? it's a script :P |
|
12:59
🔗
|
joepie91 |
no GUI |
|
12:59
🔗
|
midas |
lol :p |
|
12:59
🔗
|
joepie91 |
I've now gotten to the point that I can give it a collection name |
|
12:59
🔗
|
joepie91 |
and it'll download all of the WARC index files in every item of that collection |
|
12:59
🔗
|
Bobby__ |
Hi, I got kicked as Bobby and Bobby_, so now I'm Bobby__ ... :P |
|
12:59
🔗
|
joepie91 |
gunzip them |
|
12:59
🔗
|
joepie91 |
and chomp them into lines |
|
12:59
🔗
|
joepie91 |
next up: filtering for the lines we want and downloading the WARCs |
|
12:59
🔗
|
joepie91 |
or well, WARC segments really |
|
12:59
🔗
|
joepie91 |
hm. |
|
13:00
🔗
|
* |
joepie91 thinks |
|
13:00
🔗
|
midas |
inb4 joepie91 does imput % and disconnects from the internet |
|
13:00
🔗
|
joepie91 |
what? :P |
|
13:00
🔗
|
midas |
match all, downloads archive.org :p |
|
13:01
🔗
|
garyrh |
heh, only 20PB or so |
|
13:01
🔗
|
|
Bobby_ has quit IRC (Read error: Operation timed out) |
|
13:01
🔗
|
midas |
actually more 45PB :p |
|
13:02
🔗
|
arkiver |
20039 TB |
|
13:03
🔗
|
garyrh |
the 20pb is the deduped data iirc |
|
13:04
🔗
|
godane |
i added the domain names to the keywords of the first archivebot grab: https://archive.org/details/archiveteam_archivebot_go_001 |
|
13:04
🔗
|
arkiver |
IA has 22403 TB, 20039 TB filled, 1074 TB Free |
|
13:04
🔗
|
godane |
this maybe the best way for us to know what the hell we have archived |
|
13:04
🔗
|
arkiver |
51980 TB in total with backups |
|
13:04
🔗
|
godane |
with archivebot |
|
13:05
🔗
|
midas |
i hate symlinks in FTP's |
|
13:05
🔗
|
midas |
ftp://ftp.sunet.se/pub/databases/object-oriented/obst/current/gzip/gzip/gzip/gzip/gzip/gzip |
|
13:05
🔗
|
godane |
arkiver: how long does it take for IA to fill 1PB? |
|
13:06
🔗
|
joepie91 |
[14:00] <midas> match all, downloads archive.org :p |
|
13:06
🔗
|
joepie91 |
heh |
|
13:06
🔗
|
joepie91 |
:3 |
|
13:06
🔗
|
arkiver |
godane: depends https://archive.org/~tracey/mrtg/df-year.png |
|
13:06
🔗
|
arkiver |
sometimes a month |
|
13:06
🔗
|
arkiver |
sometimes 4 months |
|
13:07
🔗
|
midas |
depends how mad we get and start archiving websites like twitpic |
|
13:07
🔗
|
midas |
:p |
|
13:07
🔗
|
arkiver |
haha |
|
13:07
🔗
|
arkiver |
currently free space is dropping very fast |
|
13:08
🔗
|
godane |
domains added to pack 152: https://archive.org/details/archiveteam_archivebot_go_152 |
|
13:10
🔗
|
godane |
so another 1PB maybe add in december i guess |
|
13:11
🔗
|
arkiver |
probably yes |
|
13:17
🔗
|
godane |
i'm only thinking of holiday break and stuff |
|
13:18
🔗
|
godane |
i don't think people at IA want to get the call to install 1PB drive on christmas eve |
|
13:19
🔗
|
godane |
:P |
|
13:20
🔗
|
midas |
lol :p |
|
13:20
🔗
|
midas |
mostly because it's the size of a 42U rack |
|
13:21
🔗
|
midas |
not really a external drive you can carry around |
|
13:21
🔗
|
godane |
i know |
|
13:21
🔗
|
midas |
i know you know :) |
|
13:22
🔗
|
|
brayden has joined #archiveteam-bs |
|
13:35
🔗
|
joepie91 |
godane: but it's a christmas present! :) |
|
13:39
🔗
|
|
primus104 has quit IRC (Read error: Connection reset by peer) |
|
13:42
🔗
|
|
sankin has joined #archiveteam-bs |
|
13:43
🔗
|
godane |
i'm at 47k in my inbox |
|
13:45
🔗
|
godane |
what happened with this item: https://archive.org/details/archiveteam_archivebot_go_20141107000005 |
|
13:45
🔗
|
godane |
it as a dvd in it |
|
13:47
🔗
|
godane |
also some of the recent archivebot items only have json but no warc.gz |
|
13:47
🔗
|
godane |
like this one: https://archive.org/details/archiveteam_archivebot_go_20141106000004 |
|
13:48
🔗
|
godane |
some urls only have json for some reason |
|
13:49
🔗
|
|
BlueMaxim has quit IRC (Read error: Operation timed out) |
|
14:13
🔗
|
joepie91 |
Bobby__: I assume you're going to be around for a while still? |
|
14:14
🔗
|
|
sankin has quit IRC (Leaving.) |
|
14:18
🔗
|
|
sankin has joined #archiveteam-bs |
|
14:22
🔗
|
midas |
i hope the marrage isnt like tomorrow :p |
|
14:29
🔗
|
Bobby__ |
Hey Joepie91 and Midas :) No, They are planning on getting maried next summer, so I'll definetly stick around. There are many more sources I'm pulling pictures and stories from, so I can keep working quite a while longer before I can dive into the realms of hyves:P |
|
14:34
🔗
|
joepie91 |
Bobby__: alright |
|
14:34
🔗
|
joepie91 |
I'm currently working on code to basically parse out the entire hyves grab efficiently |
|
14:34
🔗
|
joepie91 |
but I've hit a bit of a snag (how does one selectively read out of gzipped files remotely) so it may take me a while |
|
14:35
🔗
|
joepie91 |
:p |
|
14:35
🔗
|
joepie91 |
also I'm writing this as a sort of educational example of "how do you automatically process a large archiveteam megawarc dump" for other projects, so that makes things a little more complicated as well since I can't do quick hacks, heh |
|
14:40
🔗
|
Bobby__ |
Haha, well, I am gratefull for everything y'all do:) |
|
14:54
🔗
|
|
SN4T14_ has joined #archiveteam-bs |
|
14:54
🔗
|
|
lysobit has quit IRC (Read error: No route to host) |
|
14:55
🔗
|
|
Zebranky_ has quit IRC (Read error: No route to host) |
|
14:55
🔗
|
|
dashcloud has quit IRC (Read error: No route to host) |
|
14:58
🔗
|
|
Zebranky has joined #archiveteam-bs |
|
14:58
🔗
|
|
Bobby_ has joined #archiveteam-bs |
|
14:58
🔗
|
|
brayden_ has joined #archiveteam-bs |
|
14:59
🔗
|
|
brayden has quit IRC (Read error: No route to host) |
|
15:00
🔗
|
|
dashcloud has joined #archiveteam-bs |
|
15:00
🔗
|
|
lysobit has joined #archiveteam-bs |
|
15:00
🔗
|
|
SN4T14__ has quit IRC (Read error: No route to host) |
|
15:02
🔗
|
|
Bobby__ has quit IRC (Ping timeout: 378 seconds) |
|
15:05
🔗
|
schbirid2 |
oh god i looked at gaia online and want to bleach my eyes now |
|
15:08
🔗
|
antomatic |
Save all the chibis! |
|
15:11
🔗
|
godane |
i'm at 272k items uploaded |
|
15:15
🔗
|
espes__ |
joepie91: yeah that's the thing |
|
15:15
🔗
|
espes__ |
we should get better at pulling useful data from archiveteam dumps |
|
15:16
🔗
|
espes__ |
oh, and can't you do all what you said with like, curl /search | jq | xargs curl | zcat | grep? :P |
|
15:16
🔗
|
espes__ |
(pull all the indexes, that is) |
|
15:17
🔗
|
espes__ |
oh right, "without hacks" :P |
|
15:18
🔗
|
joepie91 |
espes__: well, you see, the problem here is |
|
15:19
🔗
|
joepie91 |
gzip doesn't *really* support random access |
|
15:19
🔗
|
joepie91 |
and I wasn't planning on running through X times 50GB megawarc files |
|
15:19
🔗
|
joepie91 |
to find a user |
|
15:19
🔗
|
joepie91 |
lol |
|
15:19
🔗
|
midas |
CRAP |
|
15:19
🔗
|
midas |
ERROR: No space left on device (/t/_ftp.atnf.csiro.au//2014.11.ftp.atnf.csiro.au.tar) |
|
15:19
🔗
|
joepie91 |
there is supposedly a way to do random access in gzip, though |
|
15:19
🔗
|
joepie91 |
so I'm learning gzip now |
|
15:19
🔗
|
joepie91 |
lol |
|
15:19
🔗
|
joepie91 |
midas: gave it a sizehint? |
|
15:19
🔗
|
garyrh |
WHAT HAVE YOU DONE MIDAS |
|
15:20
🔗
|
midas |
joepie91: this was a big hint: Downloading 1.8 TB |
|
15:20
🔗
|
joepie91 |
midas: not what I mean |
|
15:20
🔗
|
joepie91 |
did you specify a sizehint during upload |
|
15:20
🔗
|
joepie91 |
lol |
|
15:20
🔗
|
midas |
i know what you mean |
|
15:20
🔗
|
midas |
nu |
|
15:20
🔗
|
midas |
no* |
|
15:20
🔗
|
joepie91 |
well there's your problem |
|
15:20
🔗
|
joepie91 |
:p |
|
15:20
🔗
|
midas |
no really? :P |
|
15:22
🔗
|
midas |
hm now just a way to fix it |
|
15:23
🔗
|
espes__ |
joepie91: so that's the other thing. inspecting everything with range queries on the warcs would be like, 50 million requests |
|
15:23
🔗
|
espes__ |
which is maybe a lot? |
|
15:23
🔗
|
joepie91 |
espes__: not really |
|
15:24
🔗
|
joepie91 |
if you cache the host that the file lives on so that you don't have to bother the /download/ redirector all the time, it should be less taxing than downloading the WARCs wholesale |
|
15:24
🔗
|
joepie91 |
HTTP overhead should be minimal and you're still not reading more data than if you were to download them entirely |
|
15:24
🔗
|
joepie91 |
the only thing that would happen is a minor increase in random reads, but that should be negligible |
|
15:25
🔗
|
joepie91 |
let's not forget that, as far as I understand it, the above is basically what the wayback machine does, except without the HTTP requests :P |
|
15:25
🔗
|
joepie91 |
but afaik the wayback serves pages directly from the warc.gz files |
|
15:25
🔗
|
joepie91 |
using offset reads |
|
15:25
🔗
|
joepie91 |
(which is what the CDXes are for) |
|
15:25
🔗
|
schbirid2 |
http://starfrosch.ch/2014/11/11/hot-100-download-charts/ |
|
15:25
🔗
|
joepie91 |
I'm not awfully familiar with the archive internals, but afaik this is how it all works behind the scenes |
|
15:25
🔗
|
espes__ |
I think the wayback does actually do http queries to storage |
|
15:26
🔗
|
joepie91 |
really? that'd surprise me |
|
15:26
🔗
|
joepie91 |
anyway, tl;dr I'd just be recreating the behaviour of the wayback machine with potentially less overhead |
|
15:27
🔗
|
joepie91 |
now that you mention it actually... wayback lookups are counted in the download counts for WARCs |
|
15:27
🔗
|
joepie91 |
so perhaps you're right and it just uses the public HTTP interface |
|
15:27
🔗
|
espes__ |
so it's still nearly equivalent to doing 50 million queries to the wayback :P |
|
15:28
🔗
|
joepie91 |
yes, without the overhead of the wayback |
|
15:28
🔗
|
joepie91 |
:P |
|
15:28
🔗
|
joepie91 |
I mean, don't forget that the wayback loads static assets and needs to carry out date range searches |
|
15:28
🔗
|
joepie91 |
or date proximity searches, rather |
|
15:28
🔗
|
joepie91 |
that's a good bit heavier than doing a few reqs to known offsets in a known file on a known host, heh |
|
15:29
🔗
|
joepie91 |
and I'd just be pulling HTML of specific pages, too - I don't care about the static assets at this point |
|
15:29
🔗
|
joepie91 |
I parse through the CDXes first to find items of inetrest |
|
15:29
🔗
|
joepie91 |
interest * |
|
15:29
🔗
|
espes__ |
probably only 10 mil with logic assuming most paginated requests are contiguous? |
|
15:29
🔗
|
joepie91 |
they seem to be contiguous from what I've seen so far - sorted by user, even |
|
15:29
🔗
|
joepie91 |
which I presume is because of small WARCs being packed up into a megawarc |
|
15:30
🔗
|
joepie91 |
anyway, the CDX is in the same order as the actual WARC records |
|
15:30
🔗
|
joepie91 |
and my code handles the requests in the order that they are found in the CDX |
|
15:30
🔗
|
joepie91 |
so that should minimize disk seeks and such |
|
15:31
🔗
|
joepie91 |
basically, I'm not very concerned about this taxing the archive too much |
|
15:31
🔗
|
joepie91 |
and if it does, I'm sure Jason will be screaming in my ear in a few hours |
|
15:31
🔗
|
joepie91 |
into* |
|
15:31
🔗
|
joepie91 |
:) |
|
15:32
🔗
|
espes__ |
yeah, no idea if 100qps is "a lot" |
|
15:40
🔗
|
midas |
and for a couple of hours joepie91 :P |
|
15:40
🔗
|
joepie91 |
midas: possibly :P |
|
15:40
🔗
|
joepie91 |
well |
|
15:40
🔗
|
joepie91 |
since the internet arcade thing |
|
15:41
🔗
|
joepie91 |
he can't really complain about anybody taking down the archive |
|
15:41
🔗
|
joepie91 |
heh |
|
15:41
🔗
|
joepie91 |
events that can successfully take down the archive: |
|
15:41
🔗
|
joepie91 |
* internet arcade |
|
15:41
🔗
|
joepie91 |
* fire |
|
15:41
🔗
|
|
aaaaaaaaa has joined #archiveteam-bs |
|
15:41
🔗
|
midas |
* power shortages |
|
15:41
🔗
|
midas |
very effective |
|
15:42
🔗
|
midas |
anyway, not sure how im going to rerun this |
|
15:42
🔗
|
midas |
btw you know what was strange joepie91 |
|
15:42
🔗
|
midas |
it died after 4MB |
|
15:43
🔗
|
midas |
4.36MB to be correct |
|
15:43
🔗
|
joepie91 |
midas: what do you mean? |
|
15:44
🔗
|
midas |
it only grabbed 4.3MB from the torrent and after that IA commited suicide |
|
15:44
🔗
|
joepie91 |
aside; per gzip specification, the correct value for the "OS" byte in the member header is "13" when compressing on an Acorn RISCOS system, and "5" when on Atari TOS - "Commodore" is conspicuously absent from the specification |
|
15:44
🔗
|
arkiver |
I believe a torrent creates the full size when it gets a byte |
|
15:44
🔗
|
joepie91 |
(I am not joking) |
|
15:44
🔗
|
arkiver |
so IA started craeting a 1.8 TB file |
|
15:44
🔗
|
arkiver |
after the 4.3 MB was sended |
|
15:45
🔗
|
midas |
hm |
|
15:47
🔗
|
ersi |
sendededed |
|
15:47
🔗
|
joepie91 |
ok.... |
|
15:47
🔗
|
* |
joepie91 stares |
|
15:47
🔗
|
joepie91 |
so I'm reading the gzip spec |
|
15:47
🔗
|
joepie91 |
and I ran across this |
|
15:47
🔗
|
joepie91 |
"This contains a Cyclic Redundancy Check value of the uncompressed data computed according to CRC-32 algorithm used in the ISO 3309 standard and in section 8.1.1.6.2 of ITU-T recommendation V.42. (See http://www.iso.ch for ordering ISO documents. See gopher://info.itu.ch for an online version of ITU-T V.42.)" |
|
15:47
🔗
|
joepie91 |
brb start my gopher client |
|
15:47
🔗
|
ersi |
:D |
|
15:53
🔗
|
midas |
lol |
|
15:53
🔗
|
midas |
2.00 TB (7,468 pieces @ 256.0 MiB) |
|
15:55
🔗
|
midas |
11 days of uploading |
|
15:55
🔗
|
midas |
diz gonna be fun |
|
15:57
🔗
|
godane |
i was no sure you could set it to 256mb |
|
16:11
🔗
|
arkiver |
midas: how did you make it derive right? |
|
16:12
🔗
|
arkiver |
derive well* |
|
16:23
🔗
|
|
aaaaaaaaa has quit IRC (Leaving) |
|
16:25
🔗
|
|
botpie91 has joined #archiveteam-bs |
|
16:30
🔗
|
|
aaaaaaaaa has joined #archiveteam-bs |
|
16:35
🔗
|
schbirid2 |
https://www.google.com/search?q=withgoogle.com |
|
17:00
🔗
|
godane |
i'm starting to download aol files again |
|
17:02
🔗
|
godane |
we can brute force gaia online it looks like: http://www.gaiaonline.com/forum/f.889/ |
|
17:03
🔗
|
|
Bobby_ has quit IRC (Read error: Operation timed out) |
|
17:04
🔗
|
godane |
www.gaiaonline.com/forum/t.79246943/ |
|
17:04
🔗
|
godane |
it redirects the page the real page link in there index: http://www.gaiaonline.com/forum/my-little-pony/my-little-pony-rules-guidelines-updated-9-5-2012/t.79246943/ |
|
17:05
🔗
|
schbirid2 |
very awesome http://listen.hatnote.com/#fr,en |
|
17:06
🔗
|
|
Bobby_ has joined #archiveteam-bs |
|
17:32
🔗
|
|
primus104 has joined #archiveteam-bs |
|
17:51
🔗
|
SmileyG |
Soooo anyone looked at grabbing kickstarters... |
|
17:58
🔗
|
|
primus104 has quit IRC (Leaving.) |
|
18:04
🔗
|
Bobby_ |
As in, the website? |
|
18:09
🔗
|
SketchCow |
An interesting idea. |
|
18:11
🔗
|
Bobby_ |
Sorry for asking so many questions all the time, but what would that mean exactly, grabbing kickstarters?:) |
|
18:12
🔗
|
joepie91 |
Bobby_: it's fine - I suspect he means archiving pages of kickstarter campaigns |
|
18:12
🔗
|
joepie91 |
cc SmileyG |
|
18:13
🔗
|
joepie91 |
which seems like a good idea to me given the tendency for them to vanish and/or be changed and/or for them to fail |
|
18:16
🔗
|
|
kyan has joined #archiveteam-bs |
|
18:17
🔗
|
Bobby_ |
Thank you joepie, the more I read here, the more this is starting to interest me, but there is so much jargon, that I continuesly have questions:P |
|
18:20
🔗
|
|
kyan has quit IRC (Client Quit) |
|
18:22
🔗
|
|
kyan has joined #archiveteam-bs |
|
18:23
🔗
|
kyan |
Looks like the chat logs at http://badcheese.com/~steve/atlogs/ have died |
|
18:25
🔗
|
aaaaaaaaa |
swebb used to have a log bot in here, but it disappeared a few days ago. However, there is another logging service. |
|
18:26
🔗
|
aaaaaaaaa |
http://archive.fart.website/bin/irclogger_logs |
|
19:02
🔗
|
|
mistym has joined #archiveteam-bs |
|
19:03
🔗
|
|
Bobby__ has joined #archiveteam-bs |
|
19:04
🔗
|
joepie91 |
Bobby__: feel free to ask, as long as it's something that you can't trivially Google all will be fine :) |
|
19:04
🔗
|
joepie91 |
and the jargon is actually not all that extensive, most things mean exactly what they sound like |
|
19:05
🔗
|
joepie91 |
they're just... not always conventional things to do, heh |
|
19:05
🔗
|
joepie91 |
eg. "let's download Hyves" is something that'd make you wonder whether there's some kind of jargon there, but it meant exactly that :P |
|
19:06
🔗
|
joepie91 |
aside; looks like I figured out the gzip issue, I was over-engineering, heh |
|
19:06
🔗
|
joepie91 |
apparently you can just feed an incomplete chunk of a gzip file into gzip, and it'll just decompress it and barf out a warning |
|
19:06
🔗
|
|
Bobby_ has quit IRC (Ping timeout: 504 seconds) |
|
19:06
🔗
|
joepie91 |
saying that it's missing data |
|
19:06
🔗
|
Bobby__ |
Thanks Joepie, my internet drops every once in a while, thats why I relog as often as I do. But yeah, downloading hyves sounds pretty weird:P |
|
19:07
🔗
|
joepie91 |
heh, your internet is nowhere near as bad as mine was back on KPN... |
|
19:07
🔗
|
* |
joepie91 sighs |
|
19:07
🔗
|
joepie91 |
couldn't stay connected to IRC for more than 5 minutes :| |
|
19:08
🔗
|
Bobby__ |
Wow, okay, yeah, that would suck:P |
|
19:09
🔗
|
joepie91 |
Bobby__: the worst part was that it was a FttH (fiber to the home) connection we were paying 75 euro a month for... |
|
19:09
🔗
|
joepie91 |
anyway. KPN is awful in general |
|
19:09
🔗
|
joepie91 |
or well, their network / customer support is |
|
19:09
🔗
|
Bobby__ |
Haha, true that:P |
|
19:10
🔗
|
joepie91 |
XS4ALL now, which is waaaaaaay better, despite being technically part of KPN |
|
19:13
🔗
|
Bobby__ |
I believe this is an international problem, providers not living up to expectations and promises. |
|
19:20
🔗
|
joepie91 |
Bobby__: yep, but KPN is a special kind of terrible |
|
19:20
🔗
|
joepie91 |
:P |
|
19:30
🔗
|
Bobby__ |
whaha:P |
|
19:37
🔗
|
|
primus104 has joined #archiveteam-bs |
|
19:49
🔗
|
|
aaaaaaaaa has quit IRC (Leaving) |
|
19:55
🔗
|
|
aaaaaaaaa has joined #archiveteam-bs |
|
20:00
🔗
|
SmileyG |
hell it's bigger than international, it's a universal problem |
|
20:01
🔗
|
joepie91 |
SmileyG: did you hear about SaturnusCom?! they raised their prices by 20 krimoleans last month! |
|
20:01
🔗
|
joepie91 |
:) |
|
20:02
🔗
|
SmileyG |
:D |
|
20:02
🔗
|
garyrh |
Goddamn Vogons. |
|
20:03
🔗
|
SmileyG |
you read one bit of poetry.... |
|
20:06
🔗
|
|
ex-parro1 has joined #archiveteam-bs |
|
20:39
🔗
|
|
BlueMaxim has joined #archiveteam-bs |
|
20:56
🔗
|
kyan |
aaaaaaaaa, thanks :) |
|
20:58
🔗
|
|
Bobby_ has joined #archiveteam-bs |
|
21:05
🔗
|
|
Bobby__ has quit IRC (Ping timeout: 504 seconds) |
|
21:17
🔗
|
joepie91 |
okay guys |
|
21:17
🔗
|
joepie91 |
for a change in tune |
|
21:18
🔗
|
joepie91 |
for once, a shutdown that doesn't suck: https://blog.andyet.com/2014/11/11/goodbye-andbang |
|
21:44
🔗
|
|
mistym has quit IRC (Remote host closed the connection) |
|
21:49
🔗
|
|
sankin has quit IRC (Leaving.) |
|
21:57
🔗
|
|
garyrh has quit IRC (Remote host closed the connection) |
|
21:58
🔗
|
|
cbb has joined #archiveteam-bs |
|
22:20
🔗
|
|
cbb has quit IRC (Quit: Nettalk6 - www.ntalk.de) |
|
22:22
🔗
|
|
garyrh has joined #archiveteam-bs |
|
22:58
🔗
|
|
Bobby__ has joined #archiveteam-bs |
|
23:00
🔗
|
|
Bobby__ has quit IRC (Client Quit) |
|
23:00
🔗
|
|
aaaaaaaaa has quit IRC (Read error: Operation timed out) |
|
23:04
🔗
|
|
Bobby_ has quit IRC (Ping timeout: 378 seconds) |
|
23:11
🔗
|
|
Kazzy has quit IRC (Ping timeout: 265 seconds) |
|
23:11
🔗
|
|
Kazzy has joined #archiveteam-bs |
|
23:13
🔗
|
Kazzy |
zzz, shitty cheap hosts |
|
23:15
🔗
|
|
kyan has quit IRC (Quit: This computer has gone to sleep) |
|
23:15
🔗
|
joepie91 |
Kazzy: RamNode NL, yes? |
|
23:16
🔗
|
Kazzy |
that's the one |
|
23:16
🔗
|
Kazzy |
multiple times a day, disconnects for a few minutes, comes back |
|
23:16
🔗
|
joepie91 |
Kazzy: not seeing any connectivity issues to my ramnode NL VPS whatsoever... |
|
23:16
🔗
|
joepie91 |
and haven't seen any since I got it |
|
23:16
🔗
|
Kazzy |
i'm connected again now, i just timed out though |
|
23:16
🔗
|
|
kyan has joined #archiveteam-bs |
|
23:16
🔗
|
Kazzy |
control panel was inaccessbile for me at the time, too |
|
23:16
🔗
|
joepie91 |
thats rather strange... |
|
23:17
🔗
|
joepie91 |
anyway, hardly related to a host being cheap :P |
|
23:17
🔗
|
Kazzy |
http://status.ramnode.com/1281198 |
|
23:18
🔗
|
joepie91 |
Kazzy: seems to be isolated to that particular node? |
|
23:18
🔗
|
|
aaaaaaaaa has joined #archiveteam-bs |
|
23:19
🔗
|
Kazzy |
there's a lot i can see from http://status.ramnode.com/ currently showing as 'down', the graphs on those ones have similar downtime to mine |
|
23:19
🔗
|
Kazzy |
not all of them though |
|
23:19
🔗
|
Kazzy |
maybe i'll complain tomorrow, see if i can get put on a better node |
|
23:20
🔗
|
joepie91 |
huh, weird |
|
23:20
🔗
|
joepie91 |
that's a very, very strange graph |
|
23:20
🔗
|
joepie91 |
or well |
|
23:20
🔗
|
joepie91 |
"graph" |
|
23:21
🔗
|
joepie91 |
anyway, shit happens |
|
23:21
🔗
|
Kazzy |
'green line with specks of kazzy being sad' |
|
23:22
🔗
|
joepie91 |
haha |
|
23:22
🔗
|
joepie91 |
redundancy! :P |
|
23:22
🔗
|
Kazzy |
whatever anyway, i paid $15 for a year, I'm not too bothred |
|
23:22
🔗
|
Kazzy |
bothered* |
|
23:23
🔗
|
joepie91 |
lol |
|
23:23
🔗
|
joepie91 |
Kazzy: honestly, unless you pay >$100 a month, you're probably not going to find any kind of server or VM with flawless network :P |
|
23:24
🔗
|
joepie91 |
network dips happen on shared infra, as long as they're not a regular occurrence all is fine |
|
23:24
🔗
|
joepie91 |
(I do have certain offenders in mind for that 'regular occurrence' thing...) |
|
23:29
🔗
|
|
kyan has quit IRC (Quit: This computer has gone to sleep) |
|
23:30
🔗
|
|
kyan has joined #archiveteam-bs |
|
23:39
🔗
|
kyan |
I'm not impressed... YouTube says they will be deleting my messages, and to download the archive they make available |
|
23:40
🔗
|
kyan |
It includes a CSV file supposedly containing my received messages. It's empty. |
|
23:46
🔗
|
xmc |
A+ |
|
23:46
🔗
|
xmc |
that kind of shit is why i always turn on all email notifications that I can |
|
23:48
🔗
|
kyan |
yeah I did thank goodness, but I want my f*cking CSV anyway. This emoji middle finger goes out to anyone from youtube who is reading this. 🖕 |
|
23:49
🔗
|
SmileyG |
D: |
|
23:58
🔗
|
midas |
for some strange reason people who work for yahoo, youtube etc dont join this channel that often :< |
|
23:58
🔗
|
midas |
well, the people who work there do join sometimes |
|
23:58
🔗
|
midas |
that came out wrong |