#archiveteam-bs 2012-11-12,Mon

↑back Search

Time	Nickname	Message
00:50 ^🔗	DFJustin	in "how the hell does it still exist" news, http://www.mysimon.com/
01:09 ^🔗	godane	uploaded and checked it: http://archive.org/details/laptops-manuals-dump-from-tim.id.au-20121111
06:59 ^🔗	SketchCow	Back once again
06:59 ^🔗	SketchCow	Off to the UK Tuesday morning
12:38 ^🔗	godane	i think i got sometime from way back before archvieteam was around
12:38 ^🔗	godane	its called digpicz
12:38 ^🔗	godane	it was to digg for pictures site
12:39 ^🔗	godane	no warc but i have (maybe) full mirror of site after it was shutdown
12:40 ^🔗	godane	including letter to them from digg.com to not use digg as part of there site
12:46 ^🔗	SmileyG	lol
13:32 ^🔗	godane	uploaded volume 20 of smart computing or year 2009
13:32 ^🔗	godane	example url: http://archive.org/details/smartcomputing-magazine-v20i1
14:32 ^🔗	godane	item is going to be called BBC_Schools_Progs_VHSRip
16:02 ^🔗	balrog	alard: you're probably aware but standalone warctozip chokes on many warcs and online warctozip is down :/
16:06 ^🔗	alard	balrog: Does it? (The standalone thing might be an older version than the online version.) Is it? underscor maintains warctozip.archive.org. http://warctozip.herokuapp.com/ is an alternative.
16:06 ^🔗	balrog	I tried a warc from mobileme with the standalone one and it didn't work
16:06 ^🔗	balrog	http://ia601401.us.archive.org/tarview.php?tar=/27/items/mobileme-hero-1341176093/mobileme-full-1341176093.tar&file=full-1341176093%2Fc%2Fca%2Fcal%2Fcalhoun%2Fhomepage.mac.com%2Fhomepage.mac.com-calhoun.warc.gz
16:07 ^🔗	balrog	http://warctozip.herokuapp.com/http://ia601401.us.archive.org/tarview.php?tar=/27/items/mobileme-hero-1341176093/mobileme-full-1341176093.tar&file=full-1341176093%2Fc%2Fca%2Fcal%2Fcalhoun%2Fhomepage.mac.com%2Fhomepage.mac.com-calhoun.warc.gz returns an internal server error
16:08 ^🔗	alard	And it is a valid warc.gz? (tarview.php doesn't always work.)
16:09 ^🔗	balrog	the standalone one is at https://github.com/alard/warctozip
16:09 ^🔗	balrog	how do I verify that it's valid?
16:09 ^🔗	balrog	un-gzipping it and looking at it with less looks ok
16:11 ^🔗	alard	http://warctozip.herokuapp.com/46875685376-46888852316/homepage.mac.com-calhoun.zip/http://archive.org/download/mobileme-hero-1341176093/mobileme-full-1341176093.tar
16:11 ^🔗	balrog	well that's a 62.5gb file
16:13 ^🔗	balrog	it's possible that warc is not fully compliant
16:13 ^🔗	alard	It isn't if you include the range. E.g.: curl -L --range 46875685376-46888852316 http://archive.org/download/mobileme-hero-1341176093/mobileme-full-1341176093.tar
16:14 ^🔗	balrog	is there a way to verify a warc? some better error reporting would be nice
16:15 ^🔗	balrog	"bad prefix on WARC version header" hmm
16:15 ^🔗	alard	Heh, yes. Error reporting is not for lazy people. :)
16:15 ^🔗	balrog	"version field is not known (0.17,1.0,0.18)"
16:15 ^🔗	balrog	hmmmmmm
16:16 ^🔗	balrog	aha
16:16 ^🔗	balrog	line endings :/
16:17 ^🔗	alard	I've just tried the warc-proxy, it can't load the thing either. (But that may be since it shares the same code.) There's apparently something wrong with one of the WARC headers.
16:17 ^🔗	balrog	no
16:17 ^🔗	balrog	the issue is line endings
16:17 ^🔗	balrog	gunzip and look at each file in vim
16:17 ^🔗	balrog	this warc and another warc
16:18 ^🔗	alard	Any line I should look for?
16:18 ^🔗	balrog	any line, from the beginning
16:19 ^🔗	alard	That looks fine, I think. If I remember correctly the line endings should be \r\n in the WARC headers.
16:19 ^🔗	balrog	hmm, yeah they ARE \r\n
16:20 ^🔗	balrog	the errors look like this:
16:20 ^🔗	balrog	[('ignored line', 'software: Wget/1.13.4.59-2b1dd (darwin9.8.0)\r\n'), ('ignored line', 'format: WARC File Format 1.0\r\n'), ('version field is not known (0.17,1.0,0.18)', 'WARC_ISO_28500_version1_latestdraft.pdf'), ('bad prefix on WARC version header', 'conformsTo: http://bibnum.bnf.fr/')]
16:20 ^🔗	balrog	warc errors at /Users/me/Downloads/homepage.mac.com-calhoun.warc:405
16:22 ^🔗	balrog	wait, Content-Length: 0?
16:23 ^🔗	alard	Hmm, that's wrong.
16:23 ^🔗	balrog	the rest of the warc looks fine
16:24 ^🔗	alard	The Content-Length is always 0 (in the WARC headers, not in the HTTP headers).
16:24 ^🔗	balrog	another WARC I'm looking at has non-zero Content-Lengths in WARC headers
16:24 ^🔗	balrog	http://ia601401.us.archive.org/tarview.php?tar=/25/items/mobileme-hero-mobileme-hero-1331593225/mobileme-hero-1331593225.tar&file=mobileme-hero-1331593225%2Fd%2Fdg%2Fdgc%2Fdgcx%2Fhomepage.mac.com%2Fhomepage.mac.com-dgcx.warc.gz
16:25 ^🔗	alard	http://warctozip.herokuapp.com/40017991168-40023049642/homepage.mac.com-dgcx.zip/http://archive.org/download/mobileme-hero-mobileme-hero-1331593225/mobileme-hero-1331593225.tar
16:25 ^🔗	alard	No problem there, it seems.
16:26 ^🔗	balrog	that warc works fine
16:26 ^🔗	balrog	the calhoun one doesn't
16:26 ^🔗	alard	No, the calhoun one is broken.
16:27 ^🔗	balrog	that's what I just said
16:27 ^🔗	alard	Yes, and I agreed. :)
16:27 ^🔗	balrog	question is how it's broken and how it can be fixed
16:27 ^🔗	balrog	since the data looks ok
16:28 ^🔗	balrog	also what's the index that warcvalid says?
16:28 ^🔗	balrog	it's certainly not line
16:28 ^🔗	alard	It might have something to do with the Wget version: Wget/1.13.4.59-2b1dd (darwin9.8.0)
16:28 ^🔗	alard	I don't know, I've never used warcvalid.
16:28 ^🔗	alard	The byte offset, perhaps?
16:29 ^🔗	alard	We do have another copy of calhoun, by the way.
16:29 ^🔗	balrog	ok
16:29 ^🔗	balrog	still, it would be nice to fix broken warcs
16:30 ^🔗	alard	Certainly, and even better to know why they exist in the first place.
16:31 ^🔗	alard	balrog: It seems that you're the source of that file.
16:32 ^🔗	balrog	probably.
16:32 ^🔗	balrog	it probably was dumped on a big-endian machine
16:32 ^🔗	balrog	I hope that's not the issue
16:34 ^🔗	balrog	it's failing a regex match
16:34 ^🔗	alard	Well, here's how Wget gets that Content-Length: it writes the response to an open file, does an fseek to the end, uses ftell to determine the length of the file.
16:34 ^🔗	alard	The regex is not the problem, I think, just a symptom.
16:35 ^🔗	alard	The Content-Length: 0 header tells the warc reader that the record does not contain any data, and that the next record starts immediately. That 'record' doesn't have a WARC-Type header, obviously, so it breaks down.
16:35 ^🔗	alard	Have you made any warcs on that system, recently?
16:36 ^🔗	alard	Do they work?
16:36 ^🔗	balrog	I haven't, that was probably the Mac G5
16:36 ^🔗	balrog	yup, that was the Mac G5
16:37 ^🔗	balrog	right now it's inaccessible but I can test on a Mac G4 probably
16:37 ^🔗	balrog	other WARCs were made on a Mac G4 by Lord_Nigh
16:37 ^🔗	balrog	(or Lord_Nightmare)
16:38 ^🔗	balrog	might be worth checking if they have the same issues
16:40 ^🔗	alard	http://ia700403.us.archive.org/30/items/archiveteam-mobileme-index/mobileme-20120817.html#randycottingham
16:40 ^🔗	alard	It seems so.
16:40 ^🔗	alard	Lots of Content-Length: 0.
16:41 ^🔗	alard	Is it possible to make a fresh WARC with the latest Wget version?
16:47 ^🔗	balrog	latest being latest git or latest release?
16:47 ^🔗	alard	Whatever is easiest. They should be the same, at least for the WARC parts.
16:48 ^🔗	balrog	once wget compiles, sure
16:49 ^🔗	balrog	it may take some time on a G4
16:51 ^🔗	SmileyG	72843264 89% 21.70kB/s 0:06:53
16:51 ^🔗	SmileyG	:/
16:51 ^🔗	SmileyG	worlds slowest connection to aus it seems :(
18:05 ^🔗	chronomex	aus only has one connection, x.25 over troposcatter packet radio
18:05 ^🔗	chronomex	a nickel a minute
18:39 ^🔗	SmileyG	D:
18:43 ^🔗	balrog	alard: I have here some warcs that were generated with wget-1.14 on debian armel
18:43 ^🔗	balrog	and they have content-length 1 for the warc headers
18:43 ^🔗	balrog	and warcvalid also chokes on them with errors
18:44 ^🔗	balrog	yup
18:44 ^🔗	balrog	wget-1.14 on debian armel spits out Content-Length: 1
18:44 ^🔗	alard	Is armel also big-endian? (Weird that it is 1, not 0.)
18:44 ^🔗	balrog	little-endian.
18:44 ^🔗	alard	Hmm.
18:45 ^🔗	balrog	yes, on ppc it writes 0
18:46 ^🔗	balrog	just tested with wget-1.14 from homebrew
18:46 ^🔗	balrog	on intel/x86_64 it works as expected
18:49 ^🔗	balrog	it's only the WARC headers that are wrong, the normal wget headers are correct
18:49 ^🔗	balrog	btw, armel means "ARM, endian little"
18:49 ^🔗	balrog	I think
18:50 ^🔗	chronomex	yep
18:51 ^🔗	balrog	anyway, this is annoying, though totally fixable in post
18:51 ^🔗	balrog	you just read the length of the headers and write the correct Content-Length
18:51 ^🔗	alard	As long as there are no warc headers inside the responses.
18:51 ^🔗	balrog	you mean that you didn't wget a site containing warcs? yeah, but then you could go looking for response headers and ignore all data within those
18:52 ^🔗	alard	Ah, yes, of course.
18:52 ^🔗	balrog	because all WARC headers start with WARC/1.0 and contain a Content-Length:
18:52 ^🔗	balrog	while HTTP headers start with HTTP/
18:52 ^🔗	balrog	and contain a valid Content-Length
18:53 ^🔗	balrog	now where is the code messing this up?
18:53 ^🔗	balrog	that's my question
18:54 ^🔗	alard	In here: http://git.savannah.gnu.org/cgit/wget.git/tree/src/warc.c
18:54 ^🔗	balrog	I know, I had that open
18:54 ^🔗	balrog	http://git.savannah.gnu.org/cgit/wget.git/tree/src/warc.c#n247 to be exact
18:55 ^🔗	alard	The extra gzip header might also be incorrect: http://git.savannah.gnu.org/cgit/wget.git/tree/src/warc.c#n196
18:55 ^🔗	balrog	doesn't asprintf return -1 on failure?
18:56 ^🔗	alard	The documentation says so.
18:56 ^🔗	balrog	well !(-1) == 0
18:57 ^🔗	alard	That might explain why there isn't an error message. But why does it go wrong?
18:58 ^🔗	balrog	the data in content_length must be corrupted
18:58 ^🔗	balrog	somewhere
18:58 ^🔗	balrog	if the asprintf doesn't complete it would just be uninitialized mem
18:59 ^🔗	balrog	but why does it spit out 0 or 1
19:00 ^🔗	balrog	ftello should be returning the appropriate value
19:20 ^🔗	balrog	alard:
19:20 ^🔗	balrog	char *content_length;
19:20 ^🔗	balrog	off_t data_len;
19:20 ^🔗	balrog	asprintf (&content_length, "%ld", data_len)
19:20 ^🔗	balrog	tell me what's wrong there.
19:21 ^🔗	balrog	or not? this is weird
19:21 ^🔗	balrog	content_length is never getting filled properly
19:21 ^🔗	balrog	data_len is correct
19:21 ^🔗	balrog	hmm, okay...
19:32 ^🔗	balrog	I don't get this
19:38 ^🔗	balrog	I changed the code to look like:
19:38 ^🔗	balrog	data_len = ftello (data_in);
19:38 ^🔗	balrog	if (asprintf (&content_length, "%ld", data_len) == -1)
19:38 ^🔗	balrog	and data_len is correct while content_length is still zero
19:45 ^🔗	balrog	alard: the issue is incorrect use of asprintf
19:45 ^🔗	balrog	you need to use "%jd", not "%ld"
19:45 ^🔗	balrog	also while you're at it, do == -1
19:45 ^🔗	SmileyG	add jack daniels!
19:46 ^🔗	balrog	alard: how to get this fixed upstream?
19:52 ^🔗	DFJustin	http://gizmodo.com/5959812/john-mcafee-wanted-for-murder
19:53 ^🔗	alard	balrog: So %jd does work?
19:53 ^🔗	balrog	yes
19:53 ^🔗	alard	Problem is, if I see that correctly, that %jd is for int while %ld is for long int.
19:54 ^🔗	alard	So %jd would break with large files on 64-bit systems?
19:54 ^🔗	balrog	j is intmax_t
19:54 ^🔗	balrog	it's whatever is the maximum width supported on the platform
19:54 ^🔗	balrog	so on 64-bit systems, it would be 64-bits
19:55 ^🔗	balrog	on 32-bit systems, it would be 32-bits
19:55 ^🔗	balrog	look it up ;)
19:55 ^🔗	alard	I may be confused with off_t, here: http://linux.die.net/man/3/fseeko
19:56 ^🔗	balrog	off_t's width may vary
19:56 ^🔗	alard	Suppose you would compile Wget with large file support, so off_t is a 64-bit number. Would %jd still work?
19:57 ^🔗	balrog	j is C99 though.
19:58 ^🔗	balrog	not sure if wget allows that
19:58 ^🔗	balrog	I think intmax_t is also C99
19:59 ^🔗	alard	Maybe it's better to use wgint (a Wget type) instead of off_t.
19:59 ^🔗	balrog	what is a wgint?
20:00 ^🔗	balrog	I mean, how wide?
20:00 ^🔗	alard	http://git.savannah.gnu.org/cgit/wget.git/tree/src/wget.h#n142
20:00 ^🔗	alard	There's a filesize function in util.c that returns a wgint.
20:00 ^🔗	balrog	yes, that's probably the way to go
20:00 ^🔗	balrog	using that function.
20:01 ^🔗	balrog	and how do you print a wgint?
20:01 ^🔗	alard	No, not the function, that needs a filename.
20:01 ^🔗	balrog	oh, doesn't work on a file pointer? :\
20:01 ^🔗	alard	But all it does is fseek to the end, then ftello, so that's the same.
20:01 ^🔗	alard	Unless there's no fseeko/ftello, but that's another case.
20:02 ^🔗	alard	number_to_static_string
20:02 ^🔗	balrog	ah
20:02 ^🔗	balrog	yes, that's the way to go here.
20:03 ^🔗	balrog	can you fix it up and get me a patch to test? I don't have the time right now
20:03 ^🔗	balrog	already wasted time debugging this :(
20:03 ^🔗	alard	Yes, I'll do that. ( http://git.savannah.gnu.org/cgit/wget.git/tree/src/utils.c#n1805 )
20:04 ^🔗	balrog	that concisely describes the problem here ;)
20:07 ^🔗	balrog	Someone's gonna have to write a script or something to fix broken warcs
20:10 ^🔗	alard	https://gist.github.com/741a969548d892183646
20:12 ^🔗	alard	Works for me. (But I have a sensible, modern OS. :)
20:13 ^🔗	balrog	It didn't work for me on modern Debian for ARM
20:13 ^🔗	balrog	;)
20:15 ^🔗	ersi	heh, "doesn't work on edgy new platform that barely no one runs"
20:15 ^🔗	ersi	:p
20:16 ^🔗	ersi	good that you guys look into these things though
20:16 ^🔗	ersi	<3<3
20:21 ^🔗	balrog	alard: is that patch against 1.14 or git?
20:22 ^🔗	alard	git, but I think it works for 1.14 as well.
20:22 ^🔗	alard	It only touches src/warc.c, and that hasn't changed (I think).
20:22 ^🔗	balrog	well that fixed the issue on ppc
20:30 ^🔗	balrog	gonna test on arm
20:30 ^🔗	balrog	how long will it take to get added upstream? :)
20:41 ^🔗	balrog	alard: yes, works on arm too
20:41 ^🔗	balrog	next steps are getting it fixed upstream, and making a warc-fixer
20:55 ^🔗	alard	balrog: Great. Shall I send the patch to the Wget people?
20:55 ^🔗	balrog	alard: please do.
20:56 ^🔗	balrog	explain that it's a pretty critical issue that breaks WARC on some platforms
20:56 ^🔗	alard	(It will take a year to get fixed. :)
20:56 ^🔗	balrog	:[
20:56 ^🔗	balrog	they won't be releasing a 1.14.1 sometime soon?
20:56 ^🔗	balrog	or do they only release .x versions to fix security bugs?
20:56 ^🔗	alard	There was a year between 1.14 and the previous version, I think.
20:57 ^🔗	alard	Don't know, we'll see.
20:57 ^🔗	alard	So on which platforms is this a problem? 32 bit? No. Platforms without large-file support?
20:58 ^🔗	balrog	not sure what's the trigger
20:58 ^🔗	balrog	I've had it happen on debian armel and on ppc mac os x 10.5.8
20:58 ^🔗	balrog	which means it probably happens on other arm platforms too, like Raspberry Pi
20:59 ^🔗	balrog	(which is armhf â hf means hardware floating point)
21:03 ^🔗	balrog	anyway how do we fix the warcs that have the wrong values?
21:04 ^🔗	ersi	What should Content-Length be in a WARC?
21:04 ^🔗	ersi	s/WARC/WARC Header/
21:05 ^🔗	balrog	it should be the length of the WARC header, correct?
21:05 ^🔗	alard	Content-Length should be the length of the record payload.
21:05 ^🔗	alard	record body, sorry.
21:05 ^🔗	balrog	this bug caused it to be garbage (0 or 1 in the cases we've looked at, but this is not guaranteed))
21:05 ^🔗	alard	So that would be the length of the HTTP headers + the HTTP body.
21:05 ^🔗	ersi	Ah, alright. I think I've seen a bug in tef's tools regarding that~ I'll investigatezor
21:05 ^🔗	chronomex	content length is how far to seek to skip the record, so headers + body sounds right to me
21:06 ^🔗	balrog	the annoying part is that we have many warcs which are broken this way, that need fixing :\|
21:06 ^🔗	chronomex	:\| \|: :\| \|:
21:06 ^🔗	ersi	Well, it's not that much of a problem - since it's actually fixable :)
21:06 ^🔗	balrog	since Lord_Nigh and I can't be the only ones who used PPC or ARM machines to participate
21:06 ^🔗	ersi	iterate through ALL the records!
21:07 ^🔗	alard	Luckily, SketchCow is going to do that anyway, for the mobileme data.
21:07 ^🔗	balrog	do what? :P
21:07 ^🔗	ersi	balrog: iterate through all the records
21:07 ^🔗	ersi	might as well fix as doing that
21:08 ^🔗	ersi	chug chug chug
21:08 ^🔗	alard	It will make it a lot slower, though.
21:08 ^🔗	ersi	It's worth it, in the long run
21:08 ^🔗	balrog	I think that warcs should be validated on the receiving end when rsynced
21:09 ^🔗	balrog	then errors like this would be caught more quickly
21:09 ^🔗	balrog	maybe only smaller warcs
21:09 ^🔗	alard	Was the Transfer-Encoding: chunked issue fixed before or during mobileme?
21:09 ^🔗	balrog	Transfer-Encoding: chunked issue? what was that?
21:09 ^🔗	ersi	I think that was during, but I'm not entirely sure
21:09 ^🔗	alard	Very early versions of the Wget code de-chunked the response but kept the Transfer-Encoding header.
21:09 ^🔗	balrog	oh, oops
21:10 ^🔗	balrog	that's stil fixable, though painfully so
21:10 ^🔗	balrog	what concerns me the most is when data is thrown away because of bugs
21:10 ^🔗	alard	The hanzo warctools fix that (warc2warc, I believe).
21:10 ^🔗	balrog	and I doubt we have that here
21:10 ^🔗	alard	It's not thrown away.
21:10 ^🔗	balrog	why not modify warc2warc to fix the Content-Length issues? :)
21:10 ^🔗	ersi	It's just a bit retardedly stored
21:11 ^🔗	ersi	There's plenty of ways to fix the WARCs, it's basically a non-issue
21:11 ^🔗	balrog	it's an issue when you're trying to extract data from them
21:11 ^🔗	alard	Although the megawarc script doesn't check the warc headers, just checks for gzip errors, so these warcs will mess up the megawarc.warc.gz's.
21:11 ^🔗	ersi	I know it's an issue, didn't say it wasn't. mark words dude :p
21:11 ^🔗	balrog	what does megawarc do, cat warcs together?
21:11 ^🔗	alard	Yes.
21:12 ^🔗	alard	It checks for gzip errors, if it's a valid gzip named .warc.gz it gets added to the megawarc, if not it's added to a tar file.
21:22 ^🔗	balrog	alard: are you sending the patch to the mailing list?
21:23 ^🔗	alard	Yes.
21:23 ^🔗	alard	Have any changes?
21:23 ^🔗	balrog	also, http://www.mail-archive.com/bug-wget@gnu.org/msg02220.html
21:23 ^🔗	balrog	no
21:23 ^🔗	balrog	but that's an interesting thing to look at and probably introduced this bug (while fixing another)
21:23 ^🔗	alard	The code works fine on many platforms, but it is apparently a problem on some PowerPC and ARM systems, and maybe others as well.
21:23 ^🔗	alard	There's a somewhat serious issue in the WARC-generating code: on some platforms (presumably the ones where off_t is not a 64-bit number) the Content-Length header at the top of each WARC record has an incorrect length. On these platforms it is sometimes 0, sometimes 1, but never the correct length. This makes the whole WARC file unreadable.
21:23 ^🔗	alard	Existing WARC files with this problem can be repaired by replacing the value of the Content-Length header with the correct value, for each WARC record in the file. The content of the WARC records is there, it's just the Content-Length header that is wrong.
21:23 ^🔗	alard	The attached patch fixes the problem in warc.c. It replaces off_t by wgint and uses the number_to_static_string function from util.c.
21:24 ^🔗	balrog	are there any other places where off_t is used "raw"?
21:24 ^🔗	balrog	or size_t
21:24 ^🔗	alard	Oh, ah, wait, it's in the warc.h.
21:25 ^🔗	balrog	funny that this compiled
21:25 ^🔗	balrog	and worked
21:28 ^🔗	alard	Fixed, good catch.
21:28 ^🔗	alard	There's a little bit of size_t in warc.c.
21:28 ^🔗	alard	But that's just used for string and buffer sizes.
21:29 ^🔗	balrog	there's a little in trunc.c but I think that's ok
21:29 ^🔗	balrog	of off_t
21:29 ^🔗	chronomex	size_t has its place ...
21:30 ^🔗	alard	Where is trunc_t?
21:30 ^🔗	balrog	yeah, size_t does
21:30 ^🔗	alard	trunc.c, sorry?
21:30 ^🔗	chronomex	do not cast size_t :(
21:30 ^🔗	balrog	util/trunc.c
21:30 ^🔗	alard	I see. (Was looking in src/)
21:30 ^🔗	chronomex	every time you truncate an integer god bukkakes on a kitten
21:31 ^🔗	ersi	awesome!
21:31 ^🔗	balrog	trunc.c is probably ok
21:32 ^🔗	balrog	since it looks like a standalone util
21:32 ^🔗	alard	I think that is everything.
21:33 ^🔗	balrog	yup
21:37 ^🔗	alard	I'm curious to see which errors this patch will introduce.
21:38 ^🔗	balrog	what sort of errors could it introduce? maybe we should test it more heavily first?
21:38 ^🔗	alard	I have no idea, but I thought off_t was a good solution.
21:39 ^🔗	alard	This wgint seems to be how they do it in the other parts of Wget, so it's probably better.
21:39 ^🔗	balrog	off_t is good but isn't portable.
21:39 ^🔗	balrog	I didn't know that, but a bit of googling indicated as much.
21:39 ^🔗	alard	Would this mean that your ARM system can't download files larger than 2GB?
21:40 ^🔗	balrog	it shouldn't mean that, hmm
21:40 ^🔗	balrog	I'm sure I was able to download 2+gb files before
21:43 ^🔗	balrog	ah
21:43 ^🔗	ersi	give it a try :)
21:43 ^🔗	balrog	getconf LFS_CFLAGS tells you what defines you need for 64-bit off_t
21:43 ^🔗	balrog	and it's -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
21:43 ^🔗	balrog	I think the issue is that the system is 32-bit and the off_t is 64-bit
21:44 ^🔗	balrog	and a long is 32 bits, not 64, typically
21:44 ^🔗	balrog	%ld means long
21:45 ^🔗	balrog	does this all make sense? :)
21:46 ^🔗	balrog	long is 32 bits on 32 bit platforms, 64 bits on 64 bit platforms
21:46 ^🔗	balrog	off_t is 64 bits on all platforms with the proper defines
21:46 ^🔗	balrog	technically, this should have been broken if wget was compiled for 32-bit on any platform (without the patch)
21:47 ^🔗	chronomex	huh.
21:47 ^🔗	chronomex	and nobody noticed?
21:47 ^🔗	alard	There aren't that many 2GB files.
21:48 ^🔗	balrog	it doesn't have anything to do with >2gb files
21:48 ^🔗	chronomex	ah
21:48 ^🔗	balrog	>2gb files will download fine
21:48 ^🔗	balrog	since that's another part of wget, and the proper flags are being set for 64-bit off_t
21:48 ^🔗	alard	Yes, but if you have a >2gb file, on a 32-bit system, the WARC record will have an incorrect Content-Length?
21:49 ^🔗	balrog	if you have anything on a 32-bit system the WARC record will have an incorrect Content-Length
21:49 ^🔗	balrog	because it's truncating
21:49 ^🔗	alard	Anything?
21:49 ^🔗	ersi	Even if the Content-length is <32-bit?
21:50 ^🔗	ersi	in value
21:50 ^🔗	alard	If it really is any size on a 32-bit system, that seems like something we should have noticed, then.
21:50 ^🔗	balrog	that's what it seemed, my tests were downloading http://google.com/ which is only one small page
21:50 ^🔗	balrog	I'm testing on i386
21:50 ^🔗	ersi	wÃ¤t, huh cool
21:51 ^🔗	alard	So it wouldn't crash, but it would print the wrong Content-Length and continue?
21:51 ^🔗	balrog	nope, it's fine on intel-32
21:51 ^🔗	balrog	wat
21:52 ^🔗	balrog	and yes, that's what it was doing
21:55 ^🔗	alard	The webshots warcs are currently being parsed by the CDX indexer, so they're fine. That implies that there is either no-one with a 32-bit system working on webshots, or that it's not a problem (for those systems).
21:56 ^🔗	alard	I think the warriors are 32-bits, even.
21:56 ^🔗	ersi	Isn't the VM 32-bit?
21:56 ^🔗	balrog	It may be that asprintf behaves differently on OS X and Linux
21:56 ^🔗	balrog	Gonna test it :)
21:56 ^🔗	balrog	I did some webshots but only on intel
21:56 ^🔗	balrog	Hmm...
21:56 ^🔗	balrog	There might be platform specific behavior of asprintf
21:56 ^🔗	alard	Or some GNU-specific fix.
21:58 ^🔗	balrog	Then Debian ARM should work.
22:00 ^🔗	alard	Yes, you'd expect that. Unless the workaround excludes ARM.
22:00 ^🔗	balrog	possible.
22:00 ^🔗	balrog	this is why I hate hacks :/
22:01 ^🔗	chronomex	hmmm fascinating hack https://code.google.com/p/linear-book-scanner/
22:02 ^🔗	balrog	alard: on ARM, I can printf an off_t only if I use %jd
22:02 ^🔗	balrog	hmm
22:03 ^🔗	balrog	or %lld. so it's a long long
22:03 ^🔗	balrog	that's the problem
22:03 ^🔗	balrog	stuff becomes "undefined behavior" here
22:05 ^🔗	alard	Somehow I find this a very tiring subject. :)
22:05 ^🔗	alard	Why doesn't it just work?!
22:05 ^🔗	balrog	http://en.wikipedia.org/wiki/Undefined_behavior â implementation-defined stuff
22:06 ^🔗	balrog	which you simply cannot rely on
22:06 ^🔗	chronomex	"undefined behavior" translates to "sends dirty pictures to your mum"
22:06 ^🔗	chronomex	keep that in mind and you'll be ok
23:57 ^🔗	dashcloud	http://www.infodocket.com/2012/11/12/gif-is-the-oxford-dictionaries-usa-2012-word-of-the-year/
23:57 ^🔗	dashcloud	so, who's worse: Network Solutions or GoDaddy?
23:58 ^🔗	chronomex	what, it took 20 years?

irclogger-viewer