#archiveteam 2013-12-30,Mon

↑back Search

Time Nickname Message
00:05 🔗 ivan` bsmith093: I've got no upstream to upload a grab of this
00:06 🔗 ivan` it's > 650GB so far
02:37 🔗 ivan` it's at least 3TB
03:07 🔗 bsmith093 ok then whoever owns an ISP please grab this massive thing http://bofh.nikhef.nl/events/
03:09 🔗 ivan` eh, it's not *that* big ;)
04:15 🔗 ersi also it's already kind of on it's way
04:15 🔗 ersi OHM/ is almost completely ingested to IA and I got HAR/ laying about, "just" need to upload 'em
08:07 🔗 ivan` is last.fm still dying?
08:07 🔗 ivan` that is, really going down
08:08 🔗 xmc I'm in denial
08:28 🔗 aggrosk WAH? Last.fm dying?
08:59 🔗 arkiver bsmith093: I'm also checking your website
09:02 🔗 bsmith093 arkiver: i have completely forgotten ... what website?
09:02 🔗 arkiver bsmith093: http://bofh.nikhef.nl/events/
09:03 🔗 arkiver going really fast
09:03 🔗 arkiver around 50-100 links per second
09:03 🔗 arkiver biggest file yet is 44357368943 bytes
09:03 🔗 bsmith093 arkiver: oh right that... so what are you checking it with? I'd love to know how its going that fast
09:04 🔗 arkiver Xenu
09:04 🔗 arkiver I'm using that one now for everything I do
09:04 🔗 arkiver I'm first checking every website I'm going to download
09:04 🔗 arkiver and then download the individual links
09:05 🔗 arkiver that way archiving a website is going A LOT faster
09:05 🔗 arkiver (did full warhammeronline.com in less then 45 minutes...)
09:11 🔗 arkiver bsmith093: already discovered almost 100000 links
09:13 🔗 bsmith093 arkiver: so how do you knoe how big its going to get beforehand? with Xenu?
09:13 🔗 bsmith093 know
09:13 🔗 arkiver I see it has already discovered 100000 urls
09:13 🔗 arkiver and I can see the size of files and folders it has already crawled
09:14 🔗 arkiver biggest file yet is around 40 GB
09:14 🔗 arkiver http://bofh.nikhef.nl/events/overig/28c3-bonustracks/queergeekspanelhq.mov
09:17 🔗 arkiver 110000
09:19 🔗 arkiver 140000
09:19 🔗 arkiver wow
09:19 🔗 arkiver still rising...
09:23 🔗 arkiver bsmith093: 160000 urls...
09:39 🔗 arkiver bsmith093: almost done now
09:52 🔗 bsmith093 arkiver: if it was an ftp site, I'd just dump it into filezilla and check the queue size.
09:52 🔗 arkiver bsmith093: ah, yeah
09:52 🔗 arkiver bsmith093: still far from finished...
09:52 🔗 arkiver discovered a lot more urls
09:52 🔗 bsmith093 so much easier
09:52 🔗 arkiver over 300000 urls now
09:53 🔗 arkiver btw I can do the biggest part of the website
09:53 🔗 bsmith093 what are the specs of the thing you're running this on
09:53 🔗 arkiver the computer I'm using now?
09:54 🔗 arkiver Intel Core i5-4570
09:54 🔗 arkiver NVIDIA GeForce GTX 760
09:54 🔗 arkiver 16 GB RAM
09:54 🔗 arkiver 128 GB SDRAM
09:55 🔗 arkiver those are the most important number I think
09:55 🔗 bsmith093 ummm, whats sdram?
09:55 🔗 arkiver oh oh oops
09:55 🔗 arkiver SSD I mean
09:55 🔗 arkiver 128 GB SSD
09:56 🔗 arkiver 16 GB DDR3 SDRAM (=RAM)
09:56 🔗 bsmith093 ah, well that kicks the crap out of my Dell vostro 1710 2GB ram 320 Gb hd setup
09:56 🔗 arkiver checking an average of 81 links per second
09:56 🔗 arkiver ah yeah
09:56 🔗 arkiver for this you do need a lot of ram
09:56 🔗 arkiver I also got 17 TB of external space here
09:58 🔗 bsmith093 in *what*, a rack mounted server cluster?!?!
09:59 🔗 arkiver lol
09:59 🔗 arkiver just 6 harddrives sitting next to each other
09:59 🔗 arkiver :P
10:44 🔗 m1das storage is cheap nowadays
12:05 🔗 arkiver still going...
12:05 🔗 arkiver 800000 links now
14:01 🔗 arkiver etsi.org/deliver/ done.
14:01 🔗 arkiver Took 25 hours total
14:01 🔗 arkiver https://web.archive.org/web/20131228131746/http://www.etsi.org/deliver/
14:01 🔗 arkiver 94853 files
14:02 🔗 arkiver 31-33 GB
14:14 🔗 arkiver www.ftp-sites.org done
14:14 🔗 arkiver took 4:30 minutes
19:32 🔗 xmc arkiver: I did https://archive.org/details/etsi_standards earlier this year
19:32 🔗 * xmc nods
19:35 🔗 arkiver xmc: great! now we have both... :)
19:35 🔗 arkiver I think it is also important to have the website for those documents saved
19:35 🔗 arkiver :)
19:35 🔗 xmc yeah
21:58 🔗 arkiver bsmith093: took a little longer then expected but i think it is finished now...
22:18 🔗 arkiver bsmith093: making sure it's finished...
22:27 🔗 godane this item should be moved to cdbbsarchive collection: https://archive.org/details/cdrom-maximum-cd-2007-12
22:27 🔗 godane for a min i thought it was one of my items
22:28 🔗 godane since its copying my way of uploading of
22:28 🔗 godane *it
22:29 🔗 arkiver bsmith093: starting to calculate how big the website is... and that is EXTREMELY BIG
22:34 🔗 godane so i dd the 2007-12 of maximum pc cd
22:34 🔗 godane and the md5sum is the same
22:35 🔗 godane who owns mithrandiragain@myopera.com email address?
22:37 🔗 godane he is in archiveteam from what i can tell by his uploads

irclogger-viewer