[02:21] <balrog> https://twitter.com/amatecha/status/30187409915052851
[02:32] <mistym> balrog: 404s
[02:33] <balrog> https://twitter.com/amatecha/status/301874099150528512
[17:32] <Schbirid> any suggestion as alternative for aria2c to download 10k robots.txt files daily with many simultaneous connections (i use 20 at the moment)
[17:33] <Schbirid> aria2c likes to hang with 100% some days and i could not figure out what domain is to blame
[18:44] <ersi> Schbirid: Can't you set some sort of time out?
[18:45] <Schbirid> i did, let me check
[18:46] <Schbirid> aria2c --max-concurrent-downloads=12 --connect-timeout=5 --max-tries=2 ...  -i aria2c_download_list
[18:50] <ersi> that's only connect timeout though.. I bet you get a connection, but that the host is retarded and don't send you any data
[18:52] <Schbirid> oh indeed, thanks for that pointer!
[18:52] <ersi> np man :)
[18:54] <Schbirid> hm, not sure which timeout it would be. since this will hang until i kill it
[18:54] <Schbirid> "-t, --timeout=<SEC> Set timeout in seconds. Default: 60"
[18:55] <Schbirid> maybe i should set that minimum download speed but i have no idea when it would trigger / how it calculates
[18:55] <ersi> hmm, that sounds like a good one (re: -t)
[18:56] <Schbirid> yeah but the default sounds good too :P
[18:56] <Schbirid> i'll try it
[18:57] <ersi> dunno, 3-10 seconds seems like a reasonable maximum as well
[18:57] <ersi> I mean, that's a *slow* response.. but still plausible. 60 is very very high imo :)