[00:03] Wow.. what's your internet speed? [00:04] It's pretty sad that most of the time, upload speeds are way inferior to download speeds. [00:14] ayanami_: That wasn't the problem. I have a 1 Gb/s symmetric fibre connection at home, though this data was on a host with only a 200 Mb/s connection. Upload to IA is usually significantly slower than that, but that still wasn't the problem. The upload itself finished in two days or so. But I had issues due to the size of the dataset exceeding what IA can handle in an item, then had to move stuff [00:15] around between items, and then had issues with getting it into the Wayback Machine after the move... [00:18] eesh, heh [00:20] It's this by the way in case anyone's wondering: https://archive.org/details/files.pushshift.io_201812 [00:20] Ah, okay [00:20] Apologies [00:20] And now I'll figure out how to grab the new stuff that was uploaded in the past few months. [00:21] Lots of new useful data there, e.g. SoundCloud, Stack Exchange, and obviously new Reddit content. [00:43] *** MR9K4 has joined #archiveteam-ot [01:30] *** drcd has quit IRC (Read error: Connection reset by peer) [02:15] *** ayanami_ has quit IRC (Quit: Leaving) [02:42] *** benjins has joined #archiveteam-ot [02:42] *** BlueMax has joined #archiveteam-ot [03:32] *** GuysFree has quit IRC (Quit: Connection closed for inactivity) [04:57] *** dhyan_nat has joined #archiveteam-ot [05:26] *** BlueMax has quit IRC (Quit: Leaving) [05:50] *** BlueMax has joined #archiveteam-ot [06:13] *** Zerote has quit IRC (Ping timeout: 600 seconds) [07:03] A list of web archives used in Wikipedia https://en.wikipedia.org/wiki/Wikipedia:List_of_web_archives_on_Wikipedia [07:18] *** MrRadar2 has quit IRC (Read error: Operation timed out) [07:18] *** BnAboyZ has quit IRC (Read error: Operation timed out) [07:27] *** BnAboyZ has joined #archiveteam-ot [07:28] *** MrRadar2 has joined #archiveteam-ot [07:35] *** Zerote has joined #archiveteam-ot [08:54] *** Zerote has quit IRC (Read error: Operation timed out) [08:57] *** Zerote has joined #archiveteam-ot [08:58] *** benjinsmi has joined #archiveteam-ot [09:01] *** benjins has quit IRC (Read error: Operation timed out) [09:02] *** Odd0002_ has joined #archiveteam-ot [09:07] *** Odd0002 has quit IRC (Ping timeout: 615 seconds) [09:07] *** Odd0002_ is now known as Odd0002 [10:27] *** Verified_ has quit IRC (Remote host closed the connection) [11:51] *** kiska1 has quit IRC (Read error: Connection reset by peer) [11:52] *** kiska1 has joined #archiveteam-ot [11:52] *** Fusl sets mode: +o kiska1 [12:00] *** deathy has quit IRC (Read error: Connection reset by peer) [12:01] *** diggan has quit IRC (Read error: Connection reset by peer) [12:03] *** diggan has joined #archiveteam-ot [12:04] *** deathy has joined #archiveteam-ot [12:25] *** BlueMax has quit IRC (Quit: Leaving) [12:31] *** icedice has joined #archiveteam-ot [13:05] *** cfarquhar has quit IRC (Read error: Operation timed out) [13:10] *** Odd0002_ has joined #archiveteam-ot [13:13] *** cfarquhar has joined #archiveteam-ot [13:16] *** Odd0002 has quit IRC (Read error: Operation timed out) [13:16] *** Odd0002_ is now known as Odd0002 [13:17] *** VerifiedJ has joined #archiveteam-ot [13:35] Any idea how to best get https://vimeo.com/331540588 ? [13:35] JDownloader 2 asks for a password even though it's streamable [13:36] GetFLV gets a bunch of fragments that I guess could be put together using Avidemux [13:36] I've installed and was about try with VSO Downloader, but in order for that to work on SSL sites it has to install a root certificate, which I'm a bit worried about [13:36] I know that VSO Downloader can grab seemingly ungrabbable videos from using it before, but installing root certificates always freaked me out [13:38] Have you tried youtube-dl? I didn't check, but I'd assume it supports Vimeo. [13:39] The camera company that ordered that photo journalist ad is backtracking everything after China got upset of Tiananmen Square being in it and banned the company from - and any mention of it - from China and the Chinese Internet [13:39] JDownloader 2 probably uses youtube-dl, I would imagine [13:39] I'll give it a try [13:43] *** cfarquhar has quit IRC (Read error: Operation timed out) [13:43] *** dhyan_nat has quit IRC (Read error: Operation timed out) [13:43] Never heard that before, but I haven't used JDownloader in... a decade? [13:44] A friend managed to find the video source [13:45] VSO Downloader is for grabbing hard to get videos, sort of like GetFLV [13:50] jdownloader uses its own crap, use youtube-dl instead and you have a very good chance of being able to download almost everything on the internet [13:51] *** cfarquhar has joined #archiveteam-ot [13:51] and it indeed did download the file just fine in just a few secs http://xor.meo.ws/2f119094/cc55/4b55/8cde/3d73e73e9fb8.png [13:52] and for anyone reading this, if you're interested in downloading a live stream, avoid youtube-dl at all costs and use streamlink (https://github.com/streamlink/streamlink) instead [13:56] *** dhyan_nat has joined #archiveteam-ot [13:56] Fusl: Ok, nice [13:57] I tried using Youtube-DLG, which is a youtube-dl GUI, but that didn't find anything [13:58] https://github.com/MrS0m30n3/youtube-dl-gui/releases [13:58] Eeh, might have something to do with it not being updated for almost two years lol [13:59] Terminal > GUI anytime [13:59] I guess [14:00] I just didn't feel like spending half an hour learning how to run it properly [14:00] uh [14:01] youtube-dl [14:01] there's nothing to learn [14:01] its _literally_ just that. [14:01] Not sure about Vimeo, but on YouTube, you usually also want to use the -f option to select the best video and audio. youtube-dl's default selection isn't always very good. [14:01] Ok, nice [14:02] Some commandline tools take more effort than others [14:02] youtube-dl --continue --retries 4 --write-info-json --write-description --write-thumbnail --write-annotations --all-subs --ignore-errors -f bestvideo+bestaudio URL [14:02] as per https://www.archiveteam.org/index.php?title=YouTube (Though I'm assuming these options will also stretch to vimeo) [15:14] *** icedice2 has joined #archiveteam-ot [15:15] *** Zerote has quit IRC (Read error: Operation timed out) [15:22] *** icedice has quit IRC (Read error: Operation timed out) [15:51] *** icedice2 has quit IRC (Ping timeout: 252 seconds) [15:52] *** Zerote has joined #archiveteam-ot [15:57] *** icedice has joined #archiveteam-ot [16:55] *** Dj-Wawa has joined #archiveteam-ot [17:14] *** dhyan_nat has quit IRC (Read error: Operation timed out) [17:27] *** deathy has quit IRC () [17:27] *** deathy has joined #archiveteam-ot [17:50] *** diggan has quit IRC () [17:50] *** diggan has joined #archiveteam-ot [18:55] *** godane has quit IRC (Quit: Leaving.) [19:03] *** godane has joined #archiveteam-ot [19:41] Does anyone here happen to know how BBC's iPlayer works? Does it just require a UK IP, or is some kind of login needed? [20:00] I've seen UK IP or special DNS, work for iPlayer before [20:07] *** killsushi has joined #archiveteam-ot [20:07] the DNS way is implemented by this company, who offers a free trail period> https://unlocator.com/channel/bbc-iplayer/ [20:12] *** tsp__ has quit IRC (Remote host closed the connection) [20:26] *** godane has quit IRC (Ping timeout: 615 seconds) [20:26] *** tsp__ has joined #archiveteam-ot [20:33] *** godane has joined #archiveteam-ot [20:41] *** dhyan_nat has joined #archiveteam-ot [20:48] *** revi has quit IRC () [20:48] *** revi has joined #archiveteam-ot [21:28] *** icedice has quit IRC (Read error: Operation timed out) [21:46] *** dhyan_nat has quit IRC (Read error: Operation timed out) [22:17] login is needed, at least in my experience [22:19] I searched around a bit more earlier and read that you can simply create an account without any complications as long as you're on a UK IP (or using that DNS stuff, which is probably just routing specific domains through a proxy). [22:34] *** BlueMax has joined #archiveteam-ot [22:36] the DNS stuff doesnt use a proxy for the video stream. I remember it comes from a CDN, even in the US [22:37] Makes sense. [23:07] *** VerifiedJ has quit IRC (Quit: Leaving)