| Time |
Nickname |
Message |
|
00:05
🔗
|
ivan` |
bsmith093: I've got no upstream to upload a grab of this |
|
00:06
🔗
|
ivan` |
it's > 650GB so far |
|
02:37
🔗
|
ivan` |
it's at least 3TB |
|
03:07
🔗
|
bsmith093 |
ok then whoever owns an ISP please grab this massive thing http://bofh.nikhef.nl/events/ |
|
03:09
🔗
|
ivan` |
eh, it's not *that* big ;) |
|
04:15
🔗
|
ersi |
also it's already kind of on it's way |
|
04:15
🔗
|
ersi |
OHM/ is almost completely ingested to IA and I got HAR/ laying about, "just" need to upload 'em |
|
08:07
🔗
|
ivan` |
is last.fm still dying? |
|
08:07
🔗
|
ivan` |
that is, really going down |
|
08:08
🔗
|
xmc |
I'm in denial |
|
08:28
🔗
|
aggrosk |
WAH? Last.fm dying? |
|
08:59
🔗
|
arkiver |
bsmith093: I'm also checking your website |
|
09:02
🔗
|
bsmith093 |
arkiver: i have completely forgotten ... what website? |
|
09:02
🔗
|
arkiver |
bsmith093: http://bofh.nikhef.nl/events/ |
|
09:03
🔗
|
arkiver |
going really fast |
|
09:03
🔗
|
arkiver |
around 50-100 links per second |
|
09:03
🔗
|
arkiver |
biggest file yet is 44357368943 bytes |
|
09:03
🔗
|
bsmith093 |
arkiver: oh right that... so what are you checking it with? I'd love to know how its going that fast |
|
09:04
🔗
|
arkiver |
Xenu |
|
09:04
🔗
|
arkiver |
I'm using that one now for everything I do |
|
09:04
🔗
|
arkiver |
I'm first checking every website I'm going to download |
|
09:04
🔗
|
arkiver |
and then download the individual links |
|
09:05
🔗
|
arkiver |
that way archiving a website is going A LOT faster |
|
09:05
🔗
|
arkiver |
(did full warhammeronline.com in less then 45 minutes...) |
|
09:11
🔗
|
arkiver |
bsmith093: already discovered almost 100000 links |
|
09:13
🔗
|
bsmith093 |
arkiver: so how do you knoe how big its going to get beforehand? with Xenu? |
|
09:13
🔗
|
bsmith093 |
know |
|
09:13
🔗
|
arkiver |
I see it has already discovered 100000 urls |
|
09:13
🔗
|
arkiver |
and I can see the size of files and folders it has already crawled |
|
09:14
🔗
|
arkiver |
biggest file yet is around 40 GB |
|
09:14
🔗
|
arkiver |
http://bofh.nikhef.nl/events/overig/28c3-bonustracks/queergeekspanelhq.mov |
|
09:17
🔗
|
arkiver |
110000 |
|
09:19
🔗
|
arkiver |
140000 |
|
09:19
🔗
|
arkiver |
wow |
|
09:19
🔗
|
arkiver |
still rising... |
|
09:23
🔗
|
arkiver |
bsmith093: 160000 urls... |
|
09:39
🔗
|
arkiver |
bsmith093: almost done now |
|
09:52
🔗
|
bsmith093 |
arkiver: if it was an ftp site, I'd just dump it into filezilla and check the queue size. |
|
09:52
🔗
|
arkiver |
bsmith093: ah, yeah |
|
09:52
🔗
|
arkiver |
bsmith093: still far from finished... |
|
09:52
🔗
|
arkiver |
discovered a lot more urls |
|
09:52
🔗
|
bsmith093 |
so much easier |
|
09:52
🔗
|
arkiver |
over 300000 urls now |
|
09:53
🔗
|
arkiver |
btw I can do the biggest part of the website |
|
09:53
🔗
|
bsmith093 |
what are the specs of the thing you're running this on |
|
09:53
🔗
|
arkiver |
the computer I'm using now? |
|
09:54
🔗
|
arkiver |
Intel Core i5-4570 |
|
09:54
🔗
|
arkiver |
NVIDIA GeForce GTX 760 |
|
09:54
🔗
|
arkiver |
16 GB RAM |
|
09:54
🔗
|
arkiver |
128 GB SDRAM |
|
09:55
🔗
|
arkiver |
those are the most important number I think |
|
09:55
🔗
|
bsmith093 |
ummm, whats sdram? |
|
09:55
🔗
|
arkiver |
oh oh oops |
|
09:55
🔗
|
arkiver |
SSD I mean |
|
09:55
🔗
|
arkiver |
128 GB SSD |
|
09:56
🔗
|
arkiver |
16 GB DDR3 SDRAM (=RAM) |
|
09:56
🔗
|
bsmith093 |
ah, well that kicks the crap out of my Dell vostro 1710 2GB ram 320 Gb hd setup |
|
09:56
🔗
|
arkiver |
checking an average of 81 links per second |
|
09:56
🔗
|
arkiver |
ah yeah |
|
09:56
🔗
|
arkiver |
for this you do need a lot of ram |
|
09:56
🔗
|
arkiver |
I also got 17 TB of external space here |
|
09:58
🔗
|
bsmith093 |
in *what*, a rack mounted server cluster?!?! |
|
09:59
🔗
|
arkiver |
lol |
|
09:59
🔗
|
arkiver |
just 6 harddrives sitting next to each other |
|
09:59
🔗
|
arkiver |
:P |
|
10:44
🔗
|
m1das |
storage is cheap nowadays |
|
12:05
🔗
|
arkiver |
still going... |
|
12:05
🔗
|
arkiver |
800000 links now |
|
14:01
🔗
|
arkiver |
etsi.org/deliver/ done. |
|
14:01
🔗
|
arkiver |
Took 25 hours total |
|
14:01
🔗
|
arkiver |
https://web.archive.org/web/20131228131746/http://www.etsi.org/deliver/ |
|
14:01
🔗
|
arkiver |
94853 files |
|
14:02
🔗
|
arkiver |
31-33 GB |
|
14:14
🔗
|
arkiver |
www.ftp-sites.org done |
|
14:14
🔗
|
arkiver |
took 4:30 minutes |
|
19:32
🔗
|
xmc |
arkiver: I did https://archive.org/details/etsi_standards earlier this year |
|
19:32
🔗
|
* |
xmc nods |
|
19:35
🔗
|
arkiver |
xmc: great! now we have both... :) |
|
19:35
🔗
|
arkiver |
I think it is also important to have the website for those documents saved |
|
19:35
🔗
|
arkiver |
:) |
|
19:35
🔗
|
xmc |
yeah |
|
21:58
🔗
|
arkiver |
bsmith093: took a little longer then expected but i think it is finished now... |
|
22:18
🔗
|
arkiver |
bsmith093: making sure it's finished... |
|
22:27
🔗
|
godane |
this item should be moved to cdbbsarchive collection: https://archive.org/details/cdrom-maximum-cd-2007-12 |
|
22:27
🔗
|
godane |
for a min i thought it was one of my items |
|
22:28
🔗
|
godane |
since its copying my way of uploading of |
|
22:28
🔗
|
godane |
*it |
|
22:29
🔗
|
arkiver |
bsmith093: starting to calculate how big the website is... and that is EXTREMELY BIG |
|
22:34
🔗
|
godane |
so i dd the 2007-12 of maximum pc cd |
|
22:34
🔗
|
godane |
and the md5sum is the same |
|
22:35
🔗
|
godane |
who owns mithrandiragain@myopera.com email address? |
|
22:37
🔗
|
godane |
he is in archiveteam from what i can tell by his uploads |