[00:02] *** svchost03 has joined #archiveteam [00:03] *** svchost01 sets mode: +o svchost03 [00:04] *** svchost02 sets mode: +o svchost03 [00:04] *** svchost01 sets mode: +o svchost03 [00:06] *** svchost01 sets mode: +o svchost03 [00:07] *** svchost01 sets mode: +o svchost03 [00:08] *** svchost01 sets mode: +o svchost03 [00:10] *** svchost01 sets mode: +o svchost03 [00:11] *** svchost01 sets mode: +o svchost03 [00:12] *** svchost01 sets mode: +o svchost03 [00:14] *** svchost01 sets mode: +o svchost03 [00:15] *** svchost01 sets mode: +o svchost03 [00:17] *** svchost01 sets mode: +o svchost03 [00:18] *** svchost01 sets mode: +o svchost03 [00:19] *** svchost01 sets mode: +o svchost03 [00:21] *** svchost01 sets mode: +o svchost03 [00:22] *** svchost01 sets mode: +o svchost03 [00:23] *** svchost01 sets mode: +o svchost03 [00:24] *** svchost01 sets mode: +o svchost03 [00:25] *** svchost01 sets mode: +o svchost03 [00:26] *** svchost01 sets mode: +o svchost03 [00:27] *** svchost01 sets mode: +o svchost03 [00:28] *** svchost01 sets mode: +o svchost03 [00:29] *** svchost01 sets mode: +o svchost03 [00:30] *** svchost01 sets mode: +o svchost03 [00:31] *** svchost01 sets mode: +o svchost03 [00:32] *** svchost01 sets mode: +o svchost03 [00:33] *** svchost01 sets mode: +o svchost03 [00:34] *** svchost01 sets mode: +o svchost03 [00:35] *** svchost01 sets mode: +o svchost03 [00:36] *** svchost01 sets mode: +o svchost03 [00:38] *** svchost01 sets mode: +o svchost03 [00:38] *** JAA sets mode: -o svchost01 [00:41] *** chirlu has quit IRC (Read error: Operation timed out) [00:47] *** EdTheNerd has joined #archiveteam [00:50] *** chirlu has joined #archiveteam [00:53] *** EdTheNerd has quit IRC (Read error: Operation timed out) [00:55] *** kristian_ has quit IRC (Quit: Leaving) [01:48] *** EdTheNerd has joined #archiveteam [01:52] *** EdTheNerd has quit IRC (Read error: Operation timed out) [02:01] *** Svekla has joined #archiveteam [02:01] *** Burak has quit IRC (Read error: Connection reset by peer) [02:20] *** icedice has quit IRC (Ping timeout: 260 seconds) [02:39] *** pizzaiolo has quit IRC (Remote host closed the connection) [02:42] *** svchost01 has quit IRC (Read error: Operation timed out) [02:43] *** svchost01 has joined #archiveteam [02:47] *** EdTheNerd has joined #archiveteam [02:56] *** EdTheNerd has quit IRC (Read error: Operation timed out) [03:08] *** robink has quit IRC (Ping timeout: 246 seconds) [03:20] *** Mateon1 has quit IRC (Read error: Operation timed out) [03:20] *** Mateon1 has joined #archiveteam [03:22] *** ZexaronS has quit IRC (Read error: Connection reset by peer) [03:22] *** ZexaronS has joined #archiveteam [03:40] *** JAA sets mode: +o svchost01 [03:47] *** EdTheNerd has joined #archiveteam [03:52] *** robink has joined #archiveteam [04:00] *** EdTheNerd has quit IRC (Read error: Operation timed out) [04:20] *** sivoais_ is now known as sivoais [04:30] *** jacketcha has joined #archiveteam [04:47] *** EdTheNerd has joined #archiveteam [04:52] *** qw3rty119 has joined #archiveteam [04:56] *** qw3rty118 has quit IRC (Read error: Operation timed out) [04:58] *** EdTheNerd has quit IRC (Read error: Operation timed out) [05:38] *** zino_ has quit IRC (Ping timeout: 1212 seconds) [05:44] *** zino_ has joined #archiveteam [05:47] *** EdTheNerd has joined #archiveteam [05:56] *** EdTheNerd has quit IRC (Read error: Operation timed out) [06:05] *** BlueMaxim has quit IRC (Read error: Connection reset by peer) [06:15] *** db48x has joined #archiveteam [06:27] hm okay [06:27] so five days into the lytro cdn assets mirror, wget ran out of memory and was killed [06:28] will this always happen? should I prune the URL list down into chunks? [06:29] and when I restart this, how do I restart it so it skips all the files it downloaded already? [06:29] and, is there any risk the last WARC in use is corrupt in any way? [06:29] my launch command was: wget -x --warc-file=lfes-not-in-ia-4 --warc-cdx --warc-max-size=1G --wait=1 --random-wait -i ~/lfes-not-in-ia-4.txt [06:46] *** EdTheNerd has joined #archiveteam [06:52] well, warcat at least thinks the final WARC is fine [06:56] *** EdTheNerd has quit IRC (Read error: Operation timed out) [06:58] damnit [06:58] "WARC output does not work with --no-clobber, --no-clobber will be disabled." [06:58] it just blew away the first gig WARC [06:59] so I guess I need to just prune the URL list [07:10] *** kyounko has quit IRC (Read error: Operation timed out) [07:13] *** bwn has quit IRC (Ping timeout: 260 seconds) [07:22] *** bwn has joined #archiveteam [07:25] *** kimmer1 has joined #archiveteam [07:25] *** chirlu has quit IRC (Quit: Bye) [07:35] *** BlueMaxim has joined #archiveteam [07:44] *** jspiros has quit IRC (brb) [07:46] *** EdTheNerd has joined #archiveteam [07:50] *** jspiros has joined #archiveteam [07:54] *** EdTheNerd has quit IRC (Read error: Operation timed out) [08:47] *** EdTheNerd has joined #archiveteam [08:47] *** schbirid has joined #archiveteam [08:58] *** EdTheNerd has quit IRC (Read error: Operation timed out) [09:11] *** odemg_ has joined #archiveteam [09:13] *** odemg has quit IRC (Read error: Operation timed out) [09:14] *** kimmer12 has joined #archiveteam [09:20] *** kimmer1 has quit IRC (Read error: Operation timed out) [09:47] *** EdTheNerd has joined #archiveteam [10:02] *** EdTheNerd has quit IRC (Read error: Operation timed out) [10:30] *** odemg_ has quit IRC (Quit: Leaving) [10:36] *** kimmer1 has joined #archiveteam [10:37] *** schbirid has quit IRC (Quit: Leaving) [10:44] *** kimmer12 has quit IRC (Ping timeout: 633 seconds) [10:47] *** EdTheNerd has joined #archiveteam [10:59] *** EdTheNerd has quit IRC (Read error: Operation timed out) [11:08] *** kristian_ has joined #archiveteam [11:46] *** EdTheNerd has joined #archiveteam [11:52] *** BlueMaxim has quit IRC (Quit: Leaving) [11:54] *** EdTheNerd has quit IRC (Read error: Operation timed out) [11:55] *** Dimtree has quit IRC (Peace) [12:06] *** schbirid has joined #archiveteam [12:11] *** kristian_ has quit IRC (Quit: Leaving) [12:32] *** pizzaiolo has joined #archiveteam [12:47] *** EdTheNerd has joined #archiveteam [13:01] *** EdTheNerd has quit IRC (Read error: Operation timed out) [13:38] *** MMovie2 has joined #archiveteam [13:39] *** MMovie has quit IRC (Read error: Operation timed out) [13:47] *** EdTheNerd has joined #archiveteam [13:48] *** undedu has joined #archiveteam [13:55] I see that less than 7 days remain before the website at http://und.edu/misc/ gets deleted, and I've only been able to download around 5 effective Gigabytes with WinHTTrack, because it downloads any related websites. [13:55] There's more information here: http://und.edu/web-support/server-decommission.cfm [13:56] But I find that all subdirectories pointed there might contain the exact same data. [13:59] *** EdTheNerd has quit IRC (Read error: Operation timed out) [14:00] undedu: We grabbed that with ArchiveBot a month ago. Can you check whether we missed anything? This is our grab: https://web.archive.org/web/20171124030911/http://und.edu/misc/ [14:00] *** zgrant has joined #archiveteam [14:01] undedu: These are the pages we grabbed (recursively, with offsite links): https://pastebin.com/raw/R37AHGyV [14:07] Well, we're missing some images at least: http://und.edu/misc/ald/ vs. https://web.archive.org/web/20171124031643/http://und.edu/misc/ald/ (Probably due to the website being semi-broken HTML; it uses a backslash in the image URL.) [14:12] Some links require manual changes, e.g. /misc links to https://web.archive.org/web/20171124030911/http://und.edu/misc/screamingeagles and the links on that page don't work unless you add a trailing slash. But it seems that it captured almost everything. [14:16] For example this: https://web.archive.org/web/20171124031715/http://und.edu/misc/zhang [14:16] http://www.und.nodak.edu/instruct/zhang/ [14:16] I have around 5 Megabytes. [14:17] Add a slash and it works: https://web.archive.org/web/20171124031715/http://und.edu/misc/zhang/ [14:19] Clicking the links won't work because it's JavaScript. [14:20] I've tried to upload the remaining files of that directory here: [14:20] https://archive.org/details/und_edu_0000 [14:21] I parsed manually some XML files looking for file name properties for images. [14:21] Downloaded more than 140 images and a video. [14:22] I should upload what I get in the end to complete the mirrors. [14:23] Please do. I don't think it will be available on the Wayback Machine though. [14:24] But I think that it would probably be better to go to the physical place of the disks and ask for a copy of the data. This is how critical information gathering to edit technical/programming books was done, like the Encyclopedia of Graphics File Formats, where authors went in person to each entity that created each file format to ask for official information. [14:25] Such a copy would definitely be nice, yes. If they're willing to give you one, absolutely do upload that to IA. [14:26] This is getting a bit lengthy. Let's move to #archiveteam-bs . [14:26] Somebody who is near should go. [14:26] Or someone who can actually go. [14:27] *** JAA changes topic to: Archive Team: We're not archive.org | https://archiveteam.org/ | Lengthy discussions in #archiveteam-bs | Offtopic in #archiveteam-ot | We know about Storify [14:29] Well, you don't need to actually be at their place to acquire a copy thanks to the internet. They could even just directly upload a copy to IA if they wanted to. [14:30] But it might be a bit late to ask them for it now due to the holidays. [14:30] Anyway, #archiveteam-bs please. [14:31] (Type "/join #archiveteam-bs" to go there.) [14:43] *** kimmer12 has joined #archiveteam [14:44] *** kimmer1 has quit IRC (Read error: Connection reset by peer) [14:45] *** kimmer1 has joined #archiveteam [14:45] *** kimmer12 has quit IRC (Read error: Connection reset by peer) [14:46] *** EdTheNerd has joined #archiveteam [14:47] *** kimmer12 has joined #archiveteam [14:48] *** kimmer1 has quit IRC (Read error: Operation timed out) [14:50] *** cloudfnl has quit IRC (Read error: Operation timed out) [14:51] *** MrDignity has quit IRC (Ping timeout: 250 seconds) [14:53] *** MrDignity has joined #archiveteam [14:53] *** EdTheNerd has quit IRC (Read error: Operation timed out) [14:54] *** MrDignity has quit IRC (Remote host closed the connection) [14:57] *** MrDignity has joined #archiveteam [14:59] *** kimmer1 has joined #archiveteam [15:02] *** will has quit IRC (Quit: Goodbye) [15:05] *** will has joined #archiveteam [15:06] *** Odd0002 has quit IRC (Ping timeout: 248 seconds) [15:07] *** cloudfnl has joined #archiveteam [15:09] *** kimmer12 has quit IRC (Read error: Operation timed out) [15:22] *** odemg has joined #archiveteam [15:24] *** MMovie2 has quit IRC (Read error: Operation timed out) [15:26] *** MMovie has joined #archiveteam [15:29] *** kimmer12 has joined #archiveteam [15:38] *** kimmer1 has quit IRC (Ping timeout: 633 seconds) [15:41] *** kimmer1 has joined #archiveteam [15:43] *** kimmer13 has joined #archiveteam [15:46] *** kimmer12 has quit IRC (Ping timeout: 633 seconds) [15:48] *** EdTheNerd has joined #archiveteam [15:50] *** kimmer1 has quit IRC (Read error: Operation timed out) [15:54] *** EdTheNerd has quit IRC (Read error: Operation timed out) [16:47] *** EdTheNerd has joined #archiveteam [16:57] *** EdTheNerd has quit IRC (Ping timeout: 506 seconds) [17:41] *** undedu has quit IRC (Ping timeout: 260 seconds) [17:47] *** EdTheNerd has joined #archiveteam [17:53] *** EdTheNerd has quit IRC (Read error: Operation timed out) [18:19] *** kimmer13 has quit IRC (Ping timeout: 633 seconds) [18:43] *** odemg_ has joined #archiveteam [18:43] *** odemg has quit IRC (Read error: Operation timed out) [18:48] *** EdTheNerd has joined #archiveteam [18:58] *** Odd0002 has joined #archiveteam [19:01] *** EdTheNerd has quit IRC (Read error: Operation timed out) [19:46] *** EdTheNerd has joined #archiveteam [19:54] *** EdTheNerd has quit IRC (Read error: Operation timed out) [20:13] *** MMovie has quit IRC (Read error: Connection reset by peer) [20:41] *** kimmer1 has joined #archiveteam [20:46] *** EdTheNerd has joined #archiveteam [20:59] *** EdTheNerd has quit IRC (Read error: Operation timed out) [21:04] *** kimmer12 has joined #archiveteam [21:08] *** kimmer1 has quit IRC (Ping timeout: 633 seconds) [21:41] *** kimmer1 has joined #archiveteam [21:47] *** EdTheNerd has joined #archiveteam [21:48] *** kimmer12 has quit IRC (Read error: Operation timed out) [21:52] *** EdTheNerd has quit IRC (Ping timeout: 260 seconds) [21:59] *** kimmer12 has joined #archiveteam [22:00] *** sep332 has joined #archiveteam [22:06] *** kimmer1 has quit IRC (Read error: Operation timed out) [22:08] *** sep332 has quit IRC (Read error: Operation timed out) [22:12] *** godane has quit IRC (Read error: Operation timed out) [22:20] *** godane has joined #archiveteam [22:21] *** svchost03 sets mode: +o godane [22:28] *** godane has quit IRC (Quit: Leaving.) [22:28] *** godane has joined #archiveteam [22:29] *** svchost03 sets mode: +o godane [22:46] *** EdTheNerd has joined #archiveteam [23:03] *** EdTheNerd has quit IRC (Read error: Operation timed out) [23:05] *** BlueMaxim has joined #archiveteam [23:14] *** pizzaiolo has quit IRC (Remote host closed the connection) [23:16] *** pizzaiolo has joined #archiveteam [23:19] *** jschwart has quit IRC (Quit: Konversation terminated!) [23:22] *** kimmer1 has joined #archiveteam [23:28] *** kimmer12 has quit IRC (Ping timeout: 633 seconds) [23:47] *** EdTheNerd has joined #archiveteam [23:54] *** MorbusIff has quit IRC (Read error: Operation timed out) [23:55] *** Morbus has joined #archiveteam [23:55] *** EdTheNerd has quit IRC (Read error: Operation timed out)