[02:40] Schbirid: FOS will be available for it soon. I am currently uploading Wretch [07:56] SketchCow: excellent, i will see to get my dockstar into the network today! it would probably end up as rsync daemon to sync from. [09:46] saving warhammeronline.com in just under 60 minutes... [09:46] :) [09:46] new record!! [09:56] nice job arkiver [10:03] how?much was it ? (in size) [10:16] using a different method [10:16] not adding a website and downloading that whole website [10:16] since it is then downloading everything one by one [10:16] but I used a program that quickly discoveres all the links from a website [10:17] then I download all those links instead of the website [10:17] the website is then faster downloaded [10:37] is http://commons.wikimedia.org/ also saved by the archiveteam already? [10:43] probably backuped by the wikiteam [10:50] ah ok [11:00] arkiver: what part of it? [11:00] the text is in http://dumps.wikimedia.org/backup-index.html with some mirrors [11:00] yes [11:00] but I mean all the images and videos and so on [11:01] uploads are close to 30 TB, I spent a few months archiving them [11:01] if you find something/someone to seed the torrents, that's appreciated :) there's one per month https://archive.org/details/wikimediacommons-torrents [11:09] are you only uploading them as torrents or also as warc's? [11:10] O_o [11:10] they're uploaded as ZIP files (which contain the individual media files + XML descriptions), torrents are just a way for distribution [11:10] 30TB, thats about the storage i have in total. [11:10] no I mean are they in the wayback machine? [11:11] see https://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_XML_dumps#Media_tarballs for more info [11:11] brb [11:11] I doubt it and it wouldn't be very useful anyway, you can't download more than 100 MB per file from wayback [11:11] though the legend says you can from some machines [11:48] ?? [11:48] you can download more then 100 MB per file from the wayback machine... [12:27] arkiver: not always https://archive.org/post/1003894/wayback-machine-doesnt-support-the-range-header-aka-wget-continue-doesnt-work [12:28] Nemo_bis: I never experienced that yet... [12:28] can someone here create good scripts or little programs for windows? [12:29] arkiver: then try downloading http://web.archive.org/web/20070810113028/http://www.knams.wikimedia.org/wikimania/highquality/Wikimania05-AP1.avi and tell me what you get :) [12:31] Nemo_bis: ah yes, I see... [12:31] I did get that sometimes but never always on 100 MB [12:31] it is different everytime [12:31] but for what I learned is that it just needs to be archived again [12:31] since there was probably some kind of error in the connection at that time [12:32] ouch, that would be terrible because those videos are gone; where did you read this? [12:34] no it's just from what I tried out [12:34] I tried and tried with other links [12:34] and that is my "conclusion" [12:35] but man [12:35] maybe we should put wikimedia in the wayback machine? [12:36] that's a bit generic :) what part of it [12:36] hmm [12:37] alright if we talk about this a little later [12:37] lol [12:37] doing several things atm [12:37] and I want to have a good conversation about it [12:37] ok? [12:39] till when are you online? [15:04] the #btch project is up and running. manual script running: https://github.com/ArchiveTeam/ptch-grab [15:05] Only manual? [15:08] i need an admin to add it to projects.json please [15:17] another project ? [15:19] yahoo! is shutting down ptch. ~5 days remain. [15:21] 74k todo ? definitive number ? [15:23] chfoo: how much concurrent by ipv4 ? [15:25] nico_32: 74k should be definitive based on the list deathy gave me. i'm not sure about how concurrent threads is ok. [15:26] if possible, best advice is use a sacrificial ip address and let us know. [15:27] for ptch there was no obvious/visible rate-limiting when I did initial research/API calls. [15:28] that being said... 2 concurrent is safe...let's at least see how it goes before trying to break it [15:28] so running concurrent=2 on 4 ipv4 [15:28] got another dedicated server [15:30] Can I increase upload slots? [15:30] Concurrent uploads [15:34] the upload target is slow [15:34] ~75 kBps here [15:35] 75.98kB/s here [15:39] from Schbirid (was got klined from efnet): "hey, could someone test the speed of my jamendo vorbis album server?" [15:40] from Schbirid (was got klined from efnet): "rsync -avP 151.217.55.80::albums2 ." [15:40] s/was/who/g [15:40] from Schbirid (who got klined from efnet): "if it works, maybe someone could sync from/to fos? albums2 is the first hdd with 2TB" [15:41] from Schbirid (who got klined from efnet): "rsync -avP --dry-run 151.217.55.80::albums2 jamendo-albums/" [15:41] poke SketchCow [15:51] OK. [15:55] chfoo: the README doesn't include the instructions added in last revisions of https://github.com/ArchiveTeam/wretch-grab [15:56] Nemo_bis: noted. i'll fix it now [15:56] I guess they need to be pushed to the upstream repo? [15:56] thanks [15:57] I also noted we still require gnutls-dev[el] and openssl-dev[el], I had to install them on fedora (this used to be the most common problem, with mobileme) [15:57] so maybe that's to add too [16:11] it is openssl-dev or gnutls-dev [16:11] one is enough [16:22] hmmm [16:24] I can't make sense out of my package manager history, oh well [16:25] Nemo_bis: wait, you mean there are people that -can- make sense out of package manager history? [16:25] where do I find these mythical creatures? [16:27] :) apper is rather easy to use [16:28] but apparently I didn't install the packages I remembered, probably I'm the wrong one ;) [18:22] !! [18:22] http://www.theverge.com/2013/12/27/5248286/vdio-shut-down-by-rdio [18:25] rdio killed the vdio star [19:09] how does web,archive.org determine what imgur links they cache [19:09] yipdw: lol [20:47] zenguy_pc: I would assume it's just a crapshoot based on what their spiders reach [20:47] so popular images linked from multiple external pages are more likely