#archiveteam 2017-12-24,Sun

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
***svchost03 has joined #archiveteam
svchost01 sets mode: +o svchost03
svchost02 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
svchost01 sets mode: +o svchost03
JAA sets mode: -o svchost01
chirlu has quit IRC (Read error: Operation timed out)
[00:02]
EdTheNerd has joined #archiveteam
chirlu has joined #archiveteam
EdTheNerd has quit IRC (Read error: Operation timed out)
kristian_ has quit IRC (Quit: Leaving)
[00:47]
........... (idle for 53mn)
EdTheNerd has joined #archiveteam
EdTheNerd has quit IRC (Read error: Operation timed out)
[01:48]
Svekla has joined #archiveteam
Burak has quit IRC (Read error: Connection reset by peer)
[02:01]
.... (idle for 19mn)
icedice has quit IRC (Ping timeout: 260 seconds) [02:20]
.... (idle for 19mn)
pizzaiolo has quit IRC (Remote host closed the connection)
svchost01 has quit IRC (Read error: Operation timed out)
svchost01 has joined #archiveteam
EdTheNerd has joined #archiveteam
[02:39]
EdTheNerd has quit IRC (Read error: Operation timed out) [02:56]
robink has quit IRC (Ping timeout: 246 seconds) [03:08]
Mateon1 has quit IRC (Read error: Operation timed out)
Mateon1 has joined #archiveteam
ZexaronS has quit IRC (Read error: Connection reset by peer)
ZexaronS has joined #archiveteam
[03:20]
.... (idle for 18mn)
JAA sets mode: +o svchost01 [03:40]
EdTheNerd has joined #archiveteam [03:47]
robink has joined #archiveteam [03:52]
EdTheNerd has quit IRC (Read error: Operation timed out) [04:00]
..... (idle for 20mn)
sivoais_ is now known as sivoais [04:20]
jacketcha has joined #archiveteam [04:30]
.... (idle for 17mn)
EdTheNerd has joined #archiveteam [04:47]
qw3rty119 has joined #archiveteam
qw3rty118 has quit IRC (Read error: Operation timed out)
EdTheNerd has quit IRC (Read error: Operation timed out)
[04:52]
......... (idle for 40mn)
zino_ has quit IRC (Ping timeout: 1212 seconds) [05:38]
zino_ has joined #archiveteam
EdTheNerd has joined #archiveteam
[05:44]
EdTheNerd has quit IRC (Read error: Operation timed out) [05:56]
BlueMaxim has quit IRC (Read error: Connection reset by peer) [06:05]
db48x has joined #archiveteam [06:15]
Vito`hm okay
so five days into the lytro cdn assets mirror, wget ran out of memory and was killed
will this always happen? should I prune the URL list down into chunks?
and when I restart this, how do I restart it so it skips all the files it downloaded already?
and, is there any risk the last WARC in use is corrupt in any way?
my launch command was: wget -x --warc-file=lfes-not-in-ia-4 --warc-cdx --warc-max-size=1G --wait=1 --random-wait -i ~/lfes-not-in-ia-4.txt
[06:27]
.... (idle for 17mn)
***EdTheNerd has joined #archiveteam [06:46]
Vito`well, warcat at least thinks the final WARC is fine [06:52]
***EdTheNerd has quit IRC (Read error: Operation timed out) [06:56]
Vito`damnit
"WARC output does not work with --no-clobber, --no-clobber will be disabled."
it just blew away the first gig WARC
so I guess I need to just prune the URL list
[06:58]
***kyounko has quit IRC (Read error: Operation timed out)
bwn has quit IRC (Ping timeout: 260 seconds)
[07:10]
bwn has joined #archiveteam
kimmer1 has joined #archiveteam
chirlu has quit IRC (Quit: Bye)
[07:22]
BlueMaxim has joined #archiveteam [07:35]
jspiros has quit IRC (brb)
EdTheNerd has joined #archiveteam
jspiros has joined #archiveteam
EdTheNerd has quit IRC (Read error: Operation timed out)
[07:44]
........... (idle for 53mn)
EdTheNerd has joined #archiveteam
schbirid has joined #archiveteam
[08:47]
EdTheNerd has quit IRC (Read error: Operation timed out) [08:58]
odemg_ has joined #archiveteam
odemg has quit IRC (Read error: Operation timed out)
kimmer12 has joined #archiveteam
[09:11]
kimmer1 has quit IRC (Read error: Operation timed out) [09:20]
...... (idle for 27mn)
EdTheNerd has joined #archiveteam [09:47]
.... (idle for 15mn)
EdTheNerd has quit IRC (Read error: Operation timed out) [10:02]
...... (idle for 28mn)
odemg_ has quit IRC (Quit: Leaving) [10:30]
kimmer1 has joined #archiveteam
schbirid has quit IRC (Quit: Leaving)
[10:36]
kimmer12 has quit IRC (Ping timeout: 633 seconds)
EdTheNerd has joined #archiveteam
[10:44]
EdTheNerd has quit IRC (Read error: Operation timed out) [10:59]
kristian_ has joined #archiveteam [11:08]
........ (idle for 38mn)
EdTheNerd has joined #archiveteam [11:46]
BlueMaxim has quit IRC (Quit: Leaving)
EdTheNerd has quit IRC (Read error: Operation timed out)
Dimtree has quit IRC (Peace)
[11:52]
schbirid has joined #archiveteam [12:06]
kristian_ has quit IRC (Quit: Leaving) [12:11]
..... (idle for 21mn)
pizzaiolo has joined #archiveteam [12:32]
.... (idle for 15mn)
EdTheNerd has joined #archiveteam [12:47]
EdTheNerd has quit IRC (Read error: Operation timed out) [13:01]
........ (idle for 37mn)
MMovie2 has joined #archiveteam
MMovie has quit IRC (Read error: Operation timed out)
[13:38]
EdTheNerd has joined #archiveteam
undedu has joined #archiveteam
[13:47]
undeduI see that less than 7 days remain before the website at http://und.edu/misc/ gets deleted, and I've only been able to download around 5 effective Gigabytes with WinHTTrack, because it downloads any related websites.
There's more information here: http://und.edu/web-support/server-decommission.cfm
But I find that all subdirectories pointed there might contain the exact same data.
[13:55]
***EdTheNerd has quit IRC (Read error: Operation timed out) [13:59]
JAAundedu: We grabbed that with ArchiveBot a month ago. Can you check whether we missed anything? This is our grab: https://web.archive.org/web/20171124030911/http://und.edu/misc/ [14:00]
***zgrant has joined #archiveteam [14:00]
JAAundedu: These are the pages we grabbed (recursively, with offsite links): https://pastebin.com/raw/R37AHGyV [14:01]
Well, we're missing some images at least: http://und.edu/misc/ald/ vs. https://web.archive.org/web/20171124031643/http://und.edu/misc/ald/ (Probably due to the website being semi-broken HTML; it uses a backslash in the image URL.) [14:07]
Some links require manual changes, e.g. /misc links to https://web.archive.org/web/20171124030911/http://und.edu/misc/screamingeagles and the links on that page don't work unless you add a trailing slash. But it seems that it captured almost everything. [14:12]
undeduFor example this: https://web.archive.org/web/20171124031715/http://und.edu/misc/zhang
http://www.und.nodak.edu/instruct/zhang/
I have around 5 Megabytes.
[14:16]
JAAAdd a slash and it works: https://web.archive.org/web/20171124031715/http://und.edu/misc/zhang/
Clicking the links won't work because it's JavaScript.
[14:17]
undeduI've tried to upload the remaining files of that directory here:
https://archive.org/details/und_edu_0000
I parsed manually some XML files looking for file name properties for images.
Downloaded more than 140 images and a video.
I should upload what I get in the end to complete the mirrors.
[14:20]
JAAPlease do. I don't think it will be available on the Wayback Machine though. [14:23]
undeduBut I think that it would probably be better to go to the physical place of the disks and ask for a copy of the data. This is how critical information gathering to edit technical/programming books was done, like the Encyclopedia of Graphics File Formats, where authors went in person to each entity that created each file format to ask for official information. [14:24]
JAASuch a copy would definitely be nice, yes. If they're willing to give you one, absolutely do upload that to IA.
This is getting a bit lengthy. Let's move to #archiveteam-bs .
[14:25]
undeduSomebody who is near should go.
Or someone who can actually go.
[14:26]
***JAA changes topic to: Archive Team: We're not archive.org | https://archiveteam.org/ | Lengthy discussions in #archiveteam-bs | Offtopic in #archiveteam-ot | We know about Storify [14:27]
JAAWell, you don't need to actually be at their place to acquire a copy thanks to the internet. They could even just directly upload a copy to IA if they wanted to.
But it might be a bit late to ask them for it now due to the holidays.
Anyway, #archiveteam-bs please.
(Type "/join #archiveteam-bs" to go there.)
[14:29]
***kimmer12 has joined #archiveteam
kimmer1 has quit IRC (Read error: Connection reset by peer)
kimmer1 has joined #archiveteam
kimmer12 has quit IRC (Read error: Connection reset by peer)
EdTheNerd has joined #archiveteam
kimmer12 has joined #archiveteam
kimmer1 has quit IRC (Read error: Operation timed out)
cloudfnl has quit IRC (Read error: Operation timed out)
MrDignity has quit IRC (Ping timeout: 250 seconds)
MrDignity has joined #archiveteam
EdTheNerd has quit IRC (Read error: Operation timed out)
MrDignity has quit IRC (Remote host closed the connection)
MrDignity has joined #archiveteam
kimmer1 has joined #archiveteam
will has quit IRC (Quit: Goodbye)
will has joined #archiveteam
Odd0002 has quit IRC (Ping timeout: 248 seconds)
cloudfnl has joined #archiveteam
kimmer12 has quit IRC (Read error: Operation timed out)
[14:43]
odemg has joined #archiveteam
MMovie2 has quit IRC (Read error: Operation timed out)
MMovie has joined #archiveteam
kimmer12 has joined #archiveteam
[15:22]
kimmer1 has quit IRC (Ping timeout: 633 seconds)
kimmer1 has joined #archiveteam
kimmer13 has joined #archiveteam
kimmer12 has quit IRC (Ping timeout: 633 seconds)
EdTheNerd has joined #archiveteam
kimmer1 has quit IRC (Read error: Operation timed out)
EdTheNerd has quit IRC (Read error: Operation timed out)
[15:38]
........... (idle for 53mn)
EdTheNerd has joined #archiveteam [16:47]
EdTheNerd has quit IRC (Ping timeout: 506 seconds) [16:57]
......... (idle for 44mn)
undedu has quit IRC (Ping timeout: 260 seconds) [17:41]
EdTheNerd has joined #archiveteam [17:47]
EdTheNerd has quit IRC (Read error: Operation timed out) [17:53]
...... (idle for 26mn)
kimmer13 has quit IRC (Ping timeout: 633 seconds) [18:19]
..... (idle for 24mn)
odemg_ has joined #archiveteam
odemg has quit IRC (Read error: Operation timed out)
[18:43]
EdTheNerd has joined #archiveteam [18:48]
Odd0002 has joined #archiveteam
EdTheNerd has quit IRC (Read error: Operation timed out)
[18:58]
.......... (idle for 45mn)
EdTheNerd has joined #archiveteam [19:46]
EdTheNerd has quit IRC (Read error: Operation timed out) [19:54]
.... (idle for 19mn)
MMovie has quit IRC (Read error: Connection reset by peer) [20:13]
...... (idle for 28mn)
kimmer1 has joined #archiveteam [20:41]
EdTheNerd has joined #archiveteam [20:46]
EdTheNerd has quit IRC (Read error: Operation timed out) [20:59]
kimmer12 has joined #archiveteam
kimmer1 has quit IRC (Ping timeout: 633 seconds)
[21:04]
....... (idle for 33mn)
kimmer1 has joined #archiveteam [21:41]
EdTheNerd has joined #archiveteam
kimmer12 has quit IRC (Read error: Operation timed out)
EdTheNerd has quit IRC (Ping timeout: 260 seconds)
[21:47]
kimmer12 has joined #archiveteam
sep332 has joined #archiveteam
[21:59]
kimmer1 has quit IRC (Read error: Operation timed out)
sep332 has quit IRC (Read error: Operation timed out)
godane has quit IRC (Read error: Operation timed out)
[22:06]
godane has joined #archiveteam
svchost03 sets mode: +o godane
[22:20]
godane has quit IRC (Quit: Leaving.)
godane has joined #archiveteam
svchost03 sets mode: +o godane
[22:28]
.... (idle for 17mn)
EdTheNerd has joined #archiveteam [22:46]
.... (idle for 17mn)
EdTheNerd has quit IRC (Read error: Operation timed out)
BlueMaxim has joined #archiveteam
[23:03]
pizzaiolo has quit IRC (Remote host closed the connection)
pizzaiolo has joined #archiveteam
jschwart has quit IRC (Quit: Konversation terminated!)
kimmer1 has joined #archiveteam
[23:14]
kimmer12 has quit IRC (Ping timeout: 633 seconds) [23:28]
.... (idle for 19mn)
EdTheNerd has joined #archiveteam [23:47]
MorbusIff has quit IRC (Read error: Operation timed out)
Morbus has joined #archiveteam
EdTheNerd has quit IRC (Read error: Operation timed out)
[23:54]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)