[00:06] *** tree3 has joined #archiveteam [00:11] hi ivan`, were you able to download that YouTube channel? [00:13] *** ndiddy has quit IRC (Read error: Connection reset by peer) [00:14] also I was going to ask, if you could provide a list of those URLs? maybe we can distribute effort if there's a risk of not being complete by Saturday when they go down... these files seem to average 20 GB per hundred, so that ends up around 1 TB for the lot [00:15] I don't have access to a very fast connection at home, but from work I can fill up my 128 GB flash drive tomorrow and Friday [00:16] I could also rip the webpages for the videos quick, which would preserve description and some of the comments at least [00:16] *** ndiddy has joined #archiveteam [00:16] we could dump the webpages into #archivebot, also [00:16] it handles YouTube vids? [00:16] (heck, we could turn on —youtube-dl and get the actual videos, too) [00:16] it can, yes [00:17] please do youtube-dl through archivebot if possible [00:17] although a TB might be pushing size limits [00:17] This is really down to the wire [00:17] join #archivebot and ask [00:28] *** bwn has quit IRC (Ping timeout: 483 seconds) [00:42] tree3: I'm doing a grab too. [00:42] thanks phuzion... how many videos in your list? [00:42] tree3: Just started, I'll try to get a list built in a second [00:45] tree3: 4450 videos according to youtube-dl -citw [00:45] awesome! can you post a list somewhere please? [00:45] Building the list now :) [00:46] And I'll kick off an archivebot run for it too once I get the list built. [00:46] Great! [00:46] *** ndiddy has quit IRC (Read error: Connection reset by peer) [00:47] *** ndiddy has joined #archiveteam [00:50] *** bwn has joined #archiveteam [01:02] tree3: if you want to watch mine, they're going here http://cookie.nerds.io/jbg/yt-dl-2015-12-9/ [01:07] *** JW_work1 has joined #archiveteam [01:08] *** JW_work1 has quit IRC (Client Quit) [01:10] *** JW_work has quit IRC (Read error: Operation timed out) [01:11] achip: How many links do you have in total? [01:12] his list before had 1,059 [01:12] Oh, ok [01:12] that's the only list posted so far, but there should be 4,450 [01:13] what's the 4450 number based on? [01:13] you said 4,450 according to youtube-dl -citw [01:13] Right [01:13] and ivan` earlier cited 4,448 [01:13] Oh nice [01:13] so those are right on line with each other [01:24] phuzion, do you have a list readily available by chance? [01:24] Not yet, I ran into some problems, and it's going VERY slowly for some reason. [01:25] I'm working on it though [01:27] Trying from another machine to see if it's any faster. [01:29] Perhaps marginally slower [01:30] er, faster rather [01:31] *** ete_ has quit IRC (Remote host closed the connection) [01:39] *** JesseW has joined #archiveteam [01:40] *** philpem has quit IRC (Ping timeout: 252 seconds) [01:53] *** Boltsie__ has quit IRC (Ping timeout: 252 seconds) [01:53] *** VonGuard has quit IRC (Ping timeout: 252 seconds) [01:53] *** VonGuard has joined #archiveteam [01:53] *** Boltsie__ has joined #archiveteam [01:54] *** karissa__ has quit IRC (Ping timeout: 252 seconds) [01:54] *** karissa__ has joined #archiveteam [01:54] *** johtso has quit IRC (Ping timeout: 252 seconds) [01:54] *** JSharp___ has quit IRC (Ping timeout: 252 seconds) [01:54] *** yipdw has quit IRC (Read error: Connection reset by peer) [01:54] *** kevin has quit IRC (Ping timeout: 252 seconds) [01:54] *** _desu___ has quit IRC (Ping timeout: 252 seconds) [01:54] *** JSharp___ has joined #archiveteam [01:55] *** kevin has joined #archiveteam [01:55] *** johtso has joined #archiveteam [01:56] *** _desu___ has joined #archiveteam [01:56] *** yipdw has joined #archiveteam [01:57] *** pikhq has quit IRC (Remote host closed the connection) [02:11] *** xXx_ndidd has joined #archiveteam [02:11] *** ndiddy has quit IRC (Read error: Connection reset by peer) [02:11] tree3: 2280 IDs so far, I haven't run them through uniq or anything yet though, so there might be duplicates. [02:12] sweet [02:12] did you use --get-id? [02:12] That's literally just IDs though, I haven't started grabbing the videos yet. [02:12] Yeah [02:12] *** xXx_ndidd has quit IRC (Read error: Connection reset by peer) [02:12] *** nd1ddy has joined #archiveteam [02:12] youtube-dl -i --get-id https://youtube.com/profile-url [02:13] ahh, nice [02:13] (-i ignores simple warnings like "copyright blocked this video") [02:17] *** xXx_ndidd has joined #archiveteam [02:17] *** nd1ddy has quit IRC (Read error: Connection reset by peer) [02:18] *** pikhq has joined #archiveteam [02:40] tree3: 4100 IDs and counting [02:40] yayy [02:44] About 100 to go [02:44] according to our previous estimates [02:46] 4447 is what it seems to be stopped at [02:47] cool [02:47] pastebin please? [02:49] http://irc.teh-server.com/files/truck-vids.txt [02:49] There ya go [02:49] *** kyan has left Leaving [02:49] *** kyan has joined #archiveteam [02:49] Ty phuzion [02:50] You're quite welcome. [02:53] *** tree3 has quit IRC (Read error: Connection reset by peer) [02:54] *** tree3 has joined #archiveteam [02:56] *** xXx_ndidd has quit IRC (Read error: Connection reset by peer) [02:57] *** kyan is now known as kyan_Out2 [02:57] *** kyan_Out2 is now known as OutToLunc [03:01] *** OutToLunc is now known as kyan [03:03] *** kyan has quit IRC (Quit: Leaving) [03:19] *** remsen has joined #archiveteam [03:21] *** dashcloud has quit IRC (Read error: Operation timed out) [03:21] *** dashcloud has joined #archiveteam [03:39] *** bwn has quit IRC (Ping timeout: 483 seconds) [03:57] *** vitzli has joined #archiveteam [04:03] *** kyan has joined #archiveteam [04:06] *** dtm has quit IRC (Read error: Operation timed out) [04:13] *** dtm has joined #archiveteam [04:33] *** superkuh has quit IRC (Read error: Connection reset by peer) [04:35] *** cechk01 has joined #archiveteam [04:35] hey [04:37] Hello! [04:37] just found out about this project from /r/archiveteam [04:37] how actiive is the project? [04:37] ArchiveTeam in general? Quite active [04:37] note the 187 people in the channel [04:38] yea [04:38] i see that [04:38] so i guess to contribute i just run a warrior? [04:38] That's one way! [04:38] There are a lot of things that can be done around here, [04:38] so if that doesn't appeal to you there are plenty of other things too :) [04:38] Im an EE so im not the best at coding [04:40] but i have tons of drive space and fast internet [04:41] Awesome! What operating sytsem do you use? [04:41] windows [04:41] and linux [04:41] Ok, most of the ArchiveTeam tools that I've used are generally for Linux [04:42] so I'm not sure if they'd work on Windows [04:42] the Warrior of course should work on any OS [04:42] If you're comfortable finding your way around the command line in Linux, you could run the same scripts the Warrior does, without the VM [04:43] You can join #internetarchive.bak which is a project to mirror the Internet Archive (archive.org) [04:43] and volunteer disk space there [04:43] (depending on what you mean by tons of it) [04:43] generally I think at least 500GB free is necessary [04:44] i have 5TB free ATM [04:44] Ah! That's a good amount [04:44] yes [04:44] * kyan has around 300gb free :( [04:44] If you would like to contribute financially to archival efforts, you can donate to the Internet Archive [04:45] If you've got the equipment and access to materials, you can copy things to a computer (floppy disks, cassette/reel-to-reel audio tapes, documents, etc) and upload to the Internet Archive [04:46] If you want to research and write up details about file formats (how to read them, etc.), or how to digitize stuff, we have wikis devoted to this [04:46] ok cool [04:46] do u have the password for the wiki> [04:46] What would you like to change on it? [04:47] nothing yet [04:47] Cool :) [04:47] The password's "yahoosucks" [04:47] without the quotation marks [04:47] awesome [04:47] thx [04:47] np :) [04:48] hmm havig issues with the VM [04:49] "a breakpoint has been reached" [04:49] From your virtualization software, or once the VM has booted? [04:49] also, if you want to help scrape url shorteners, we have a large list of possible ones that need research to make sure they are still active and the particular settings needed to scrape them. Join #urlteam for details. [04:50] from virtualboc [04:50] ok [04:51] Ah, looks like that's a Windows error... [04:51] maybe someone here who knows Windows could help out? [04:51] yup [04:51] * kyan is only familiar with OS X and Debian linux [04:52] ahhh ok [04:52] Also, you may want to join #warrior to discuss it in more detail. [04:52] I'm not that familiar with Windows either, but if you follow me over there, I'll see what I can do to help. [05:04] *** cechk01 has quit IRC (Quit: Leaving) [05:09] *** BlueMaxim has quit IRC (Read error: Connection reset by peer) [05:11] *** aaaaaaaaa has quit IRC (Leaving) [05:11] *** cechk01 has joined #archiveteam [05:49] *** Sk1d has quit IRC (Ping timeout: 250 seconds) [05:58] *** Sk1d has joined #archiveteam [06:08] *** Stiletto has quit IRC () [06:10] *** bwn has joined #archiveteam [06:13] *** asdf has joined #archiveteam [06:25] *** nertzy has joined #archiveteam [06:26] *** bwn_ has joined #archiveteam [06:32] *** bwn has quit IRC (Read error: Operation timed out) [07:00] *** BlueMaxim has joined #archiveteam [07:06] *** RichardG has quit IRC (Ping timeout: 606 seconds) [07:25] *** nertzy has quit IRC (Quit: This computer has gone to sleep) [07:30] *** vitzli has quit IRC (Quit: Leaving) [08:08] *** JesseW has quit IRC (Leaving.) [08:39] *** atomotic has joined #archiveteam [08:42] *** atomotic has quit IRC (Client Quit) [08:44] *** bwn_ has quit IRC (Read error: Operation timed out) [09:00] how to upload 2gb file to archive.org? ia shows 500 error [09:04] *** vOYtEC has joined #archiveteam [09:05] *** Elegance has quit IRC (Ping timeout: 369 seconds) [09:06] *** Elegance has joined #archiveteam [09:09] Hm, are you sure it has to do with the size? I frequently upload files 20GB+, and occasionally 100gb+ with no trouble [09:09] but sometimes I get errors [09:10] if you're using a manually constructed curl command, it's important to inlude the size hint [09:10] no idea. just got '500 Server Error: Internal Server Error' again [09:10] are you uploding using a Web browser? [09:10] no, ia [09:10] Ok [09:10] What's the exact command you're running? [09:11] ia upload SOMEID FILENAME [09:12] i've uploaded some stuff earlier with the command i'm using [09:12] What is SOMEID? [09:12] Some ID :) [09:12] items id [09:12] Fine, have you made sure it's a valid ID? [09:12] yes [09:12] What is the file type? [09:13] iso image [09:13] ISO 9660 CD-ROM filesystem data [09:13] Is the ISO image encrypted, or does it have encrypted files in it? [09:13] no, it's unencrypted [09:13] Huh... [09:13] just a normal iso [09:14] could you PM me the identifier? [09:14] (just so I can see with my own eyes what it's doing) [09:14] it's doing nothing. Item History for this id is empty [09:15] Oh, not what the job's doing [09:15] but what ia upload is doing [09:15] namely, what identifier it's sending to the server [09:15] *** BlueMaxim has quit IRC (Read error: Connection reset by peer) [09:16] i'd like to keed that itemname, so i wont be sending it :P [09:16] keep it? [09:16] anyway it's ascii with dots [09:16] how long is it? [09:16] nothing special [09:16] Does it start or end with a dot? [09:16] 28 chars long [09:16] Does it have more than two dots sequentially? [09:17] no, it doesnt [09:17] hmm [09:17] Are there already files uploaded to that identifier? [09:17] pretty strange huh? [09:17] no [09:18] Yes, that's unusual [09:23] *** bwn has joined #archiveteam [09:24] useretai-, Please run to test: echo 'test' > ./test-ia-upload-2015dec9a10; ia upload SOMEID ./test-ia-upload-2015dec9a10 [09:24] and let me know whether it works? [09:24] Or more interestingly [09:24] echo 'test' > ./test-ia-upload-2015dec9a10.iso; ia upload SOMEID ./test-ia-upload-2015dec9a10.iso [09:25] just to check if there's something that's broken with your setup since earlire [09:26] erm, no i uploaded some stuff yestereday and worked pretty fine [09:26] Also bear in mind that if push comes to shove you can make a torrent file and upload that [09:26] Right, that's why I said "since earlier" [09:26] As in, maybe something has broken since your tests yesterday [09:35] no, everything works as it should. just crated another item [09:37] Was that also an ISO? [09:37] Have you tested uploading to the *same* identifier you were having trouble with earlier? [09:39] another question. my internet connection is 50/50 Mbit and according to calculations 2gb file should upload in 5 min 43 sec. currently i'm uploading via web interface 10 minutes already. why? is archive limiting my upload speed or something? [09:39] depends on your internet provider's peering [09:39] kyan: no, it wasnt iso [09:40] and wasnt same id [09:40] Ok, could you test with a file with a name ending in .iso and the same ID [09:40] (preferably not a real ISO file, but just a little text file for testing) [09:40] useretai-: just because you can use *up to* 50Mbit/s doesn't mean you always will to all destinations [09:41] *** atomotic has joined #archiveteam [09:42] yeah, i agree. but i thought that archive uses ultra high-speed connection :) [09:42] kyan: i cant. i'm uploading via web interface [09:43] earlier you said you were running ia upload SOMEID FILENAME [09:43] I thought you were talking about the syntax for the internetarchive Python module [09:43] btw web interface doesnt showed any errors on that id [09:43] which from the command line is executed in that manner [09:44] kyan: i was. but since it showed 500, i decided to try web interface ~10 min ago [09:44] Ok, so I'm asking you to test with a small text file with a name ending in .iso and the same ID, using the internetarchive Python module [09:53] *** schbirid has joined #archiveteam [09:54] yeah, i got it. but since i'm uploading using same id via web-browser i don't think it will worl [09:54] *work [09:54] if upload fail, i will try [09:55] Ok, but I won't be around much longer tongiht [09:55] no problem, i will report anyway [09:55] Ok, thanks :) [09:55] I'm curious to find out what the issue is [09:57] google found this: https://archive.org/serve/uploaded/picfixer-cats.jpg [09:57] but 'overloaded'? [10:06] *** Ghost_of_ has joined #archiveteam [10:33] *** bwn has quit IRC (Read error: Operation timed out) [10:48] *** midas has quit IRC (WeeChat 1.3) [10:50] *** trs80 has quit IRC (Remote host closed the connection) [10:54] *** midas1 has joined #archiveteam [11:20] *** bwn has joined #archiveteam [11:29] *** midas1 has quit IRC (Quit: WeeChat 1.3) [11:36] *** RichardG has joined #archiveteam [11:45] *** Morbus has quit IRC (http://www.disobey.com/) [11:48] *** Morbus has joined #archiveteam [11:56] *** midas1 has joined #archiveteam [12:01] *** Morbus has quit IRC (Quit: http://www.disobey.com/) [12:07] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [12:27] *** Morbus has joined #archiveteam [12:31] *** Morbus has quit IRC (Client Quit) [12:36] *** remsen2 has joined #archiveteam [12:42] *** remsen has quit IRC (Read error: Operation timed out) [12:53] *** K4k_ has joined #archiveteam [12:56] *** K4k has quit IRC (Ping timeout: 252 seconds) [12:57] *** Ghost_of_ has quit IRC (Quit: Leaving) [13:19] *** atomotic has joined #archiveteam [13:27] *** remsen2 has quit IRC (Leaving) [13:35] *** Laverne_ has joined #archiveteam [13:49] *** WinterFox has quit IRC (Remote host closed the connection) [13:57] *** RichardG has quit IRC (Ping timeout: 252 seconds) [13:58] *** RichardG has joined #archiveteam [14:02] *** phuzion has quit IRC (Remote host closed the connection) [14:02] *** phuzion has joined #archiveteam [14:16] *** slyphic|a is now known as slyphic [14:41] *** Ghost_of_ has joined #archiveteam [14:56] *** Start has quit IRC (Quit: Disconnected.) [15:24] *** trs80 has joined #archiveteam [15:50] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [15:56] *** Amitari has joined #archiveteam [15:56] Um, is it possible to propose sites that should be archived? [15:56] *** Start has joined #archiveteam [15:58] yes [15:58] Always [15:58] Well, I think the mainly Swedish photo community Dayviews should be archived. I'd gladly help out by hosting an archive warrior or whatever it's called. [16:03] and why do you think that should be archived? [16:04] *** Ghost_of_ has quit IRC (Quit: Leaving) [16:05] It'd old as fuck [16:05] Well, it contains a significant part of Swedish youth culture in the 00s, but I'm really surprised it's still up considering how much the user count has sunk. [16:05] I'm pretty sure Archive Team has archived non-English sites before. [16:05] oh yes [16:05] thats not the issue [16:06] so you don't have any knowledge about it being shut down a particular date? [16:07] No, but it might be shut down any time. A lot of Swedish web communities have been shut down pretty suddenly I think. [16:07] Amitari: Basically, we tend to prioritize resources toward projects that we know are actively shutting down, but if there's a lull, I suppose we could knock out Dayviews if someone took the incentive to build the scripts to archive it. [16:07] I guess the best time would be when it's shutting down. [16:08] phuzion: Oh, about that. I think there would be some problems with the technical stuff since the users can set so that only their friends can see their content. You can also set so that only registered users can see their content, but that's less of a problem since it's really easy to create an account, the registration process is available in English since they have tried to branch out before. [16:09] Wait, I think you could actually get past the first thing by using the user string of a search engine robot. [16:10] And you still can, considering that they probably don't update the actual software anymore. [16:11] Darn, it seems like it doesn't actually work anymore. :( [16:12] The public stuff and the stuff restricted to registered users could still easily be snatched though. [16:16] *** Ymgve has quit IRC () [16:22] Dayviews should at least be on the watchlist. [16:28] Amitari: Would you mind putting it on the wiki at Deathwatch then? [16:33] *** JesseW has joined #archiveteam [16:40] *** Start has quit IRC (Read error: Operation timed out) [16:41] *** Start has joined #archiveteam [16:44] phuzion: Oh, I'm allowed to do that? [16:45] Amitari: Sure, just register for an account on the wiki and add it. [16:45] Alright, will do! [16:49] *** Amitari has quit IRC (Quit: Leaving) [16:50] *** Amitari has joined #archiveteam [16:50] WHAT FORSOOTH, PRITHEE TELL ME THE SECRET WORD [16:57] *** Start has quit IRC (Quit: Disconnected.) [16:57] *** remsen has joined #archiveteam [16:57] WHAT FORSOOTH, PRITHEE TELL ME THE SECRET WORD [17:00] yahoosucks [17:00] *** JesseW has quit IRC (Leaving.) [17:04] *** Amitari has quit IRC (Leaving) [17:17] *** Start has joined #archiveteam [17:33] *** Ymgve has joined #archiveteam [17:42] *** nertzy has joined #archiveteam [17:44] *** JW_work has joined #archiveteam [17:53] *** JW_work has quit IRC (Leaving.) [18:00] *** ete_ has joined #archiveteam [18:07] *** fie has quit IRC (Read error: Operation timed out) [18:15] *** nertzy has quit IRC (Quit: This computer has gone to sleep) [18:34] *** philpem has joined #archiveteam [18:38] *** Start has quit IRC (Quit: Disconnected.) [18:47] *** JW_work has joined #archiveteam [18:48] *** Start has joined #archiveteam [19:02] *** JW_work has quit IRC (Quit: Leaving.) [19:13] *** K4k_ has quit IRC (Read error: Operation timed out) [19:19] *** Stiletto has joined #archiveteam [19:20] *** Start has quit IRC (Quit: Disconnected.) [19:31] *** Start has joined #archiveteam [19:36] *** K4k_ has joined #archiveteam [19:41] *** Start has quit IRC (Remote host closed the connection) [19:42] *** Start has joined #archiveteam [19:44] *** Start has quit IRC (Client Quit) [19:52] *** scyther has joined #archiveteam [20:04] *** K4k_ has quit IRC (Read error: Operation timed out) [20:21] *** K4k_ has joined #archiveteam [20:26] *** Start has joined #archiveteam [20:28] *** bwn has quit IRC (Read error: Operation timed out) [20:37] *** JW_work has joined #archiveteam [20:44] *** Start has quit IRC (Quit: Disconnected.) [20:50] *** Start has joined #archiveteam [20:58] *** bwn has joined #archiveteam [21:02] *** Ghost_of_ has joined #archiveteam [21:03] *** ete_ has quit IRC (Read error: Operation timed out) [21:05] *** redlob has quit IRC (Read error: Operation timed out) [21:05] *** redlob has joined #archiveteam [21:11] *** ete_ has joined #archiveteam [21:15] *** rizzzz has quit IRC (Remote host closed the connection) [21:17] *** rizzzz has joined #archiveteam [21:18] *** schbirid has quit IRC (Quit: Leaving) [21:32] *** ndiddy has joined #archiveteam [21:34] *** aaaaaaaaa has joined #archiveteam [21:34] *** swebb sets mode: +o aaaaaaaaa [21:42] *** JW_work has quit IRC (Quit: Leaving.) [21:54] kyan: reporting. everything uploaded successfully via web browser. if i will encounted same error later i will try what you told me and report accordingly [21:54] *** scyther has quit IRC (Leaving) [22:11] *** JW_work has joined #archiveteam [22:11] *** pikhq has quit IRC (Ping timeout: 252 seconds) [22:12] *** WapCapLet has quit IRC (Read error: Operation timed out) [22:13] *** Ghost_of_ has quit IRC (Quit: Leaving) [22:14] *** WapCapLet has joined #archiveteam [22:14] *** K4k_ has quit IRC (Ping timeout: 252 seconds) [22:16] *** Start has quit IRC (Quit: Disconnected.) [22:17] *** Stiletto has quit IRC () [22:20] *** mutoso has quit IRC (Remote host closed the connection) [22:20] *** K4k_ has joined #archiveteam [22:29] useretai-: Yay, congrats! [22:32] *** pikhq has joined #archiveteam [22:32] *** GLaDOS has quit IRC (Ping timeout: 252 seconds) [22:35] *** GLaDOS has joined #archiveteam [22:38] *** asdf has quit IRC (Ping timeout: 252 seconds) [22:45] *** khaoohs has quit IRC (Read error: Connection reset by peer) [22:45] *** ndiddy has quit IRC (Read error: Connection reset by peer) [22:45] *** ndiddy has joined #archiveteam [22:47] *** Stiletto has joined #archiveteam [22:50] Just found someone who has scanned dozens of issues of MacAddict [22:52] *** blergh- has joined #archiveteam [22:57] *** BlueMaxim has joined #archiveteam [22:59] *** khaoohs has joined #archiveteam [22:59] *** Stiletto has quit IRC (Read error: Connection reset by peer) [23:16] *** K4k_ has quit IRC (Read error: Operation timed out) [23:17] *** K4k_ has joined #archiveteam [23:17] *** GLaDOS has quit IRC (Ping timeout: 252 seconds) [23:19] *** GLaDOS has joined #archiveteam [23:20] *** Stiletto has joined #archiveteam [23:23] *** dtm has quit IRC (Read error: Operation timed out) [23:24] *** Ghost_of_ has joined #archiveteam [23:30] *** dtm has joined #archiveteam [23:31] did anyone happen to have fork, preferably a recent one, of ShadowVPN from GitHub? apparently it was totally nuked from the site [23:34] dashcloud: I'd check this list of forks https://github.com/clowwindy/ShadowVPN/network/members [23:35] okay- thanks [23:35] np :)