[00:09] I doubt the type of video matters for rate limiting [00:09] it's definitely not rate limiting. ie, one video every 20-40 minutes [00:10] do you get a captcha for https://www.youtube.com/watch?v=2zEIQuRg-m4 [00:33] no [00:48] I'm on a VPN and Youtube-dl works fine on there for me [00:49] Maybe you're using the same IP address as someone who's been mass-downloading or something [00:53] maybe [01:21] *** thuban3 has joined #archiveteam-ot [01:24] *** thuban2 has quit IRC (Read error: Operation timed out) [01:33] *** qw3rty_ has quit IRC (Read error: Connection reset by peer) [01:34] *** qw3rty_ has joined #archiveteam-ot [02:38] *** godane has joined #archiveteam-ot [03:07] *** kiska has quit IRC (Remote host closed the connection) [03:07] *** Flashfire has quit IRC (Remote host closed the connection) [03:08] *** kiska has joined #archiveteam-ot [03:08] *** Flashfire has joined #archiveteam-ot [03:08] *** svchfoo3 sets mode: +o kiska [03:08] *** svchfoo1 sets mode: +o kiska [04:13] *** DogsRNice has quit IRC (Read error: Connection reset by peer) [04:22] I don't ever remember seeing YT captcha me [04:23] Google search has, though [04:23] I wonder how ivan deals with it. [04:27] *** qw3rty__ has joined #archiveteam-ot [04:31] *** qw3rty_ has quit IRC (Ping timeout: 276 seconds) [04:44] use clean IPs would be the simplest [04:47] *** duh has quit IRC (Quit: http://quassel-irc.org - Chat comfortably. Anywhere.) [04:48] *** legoktm has joined #archiveteam-ot [04:57] *** cerca has quit IRC (Remote host closed the connection) [06:26] *** britmob has quit IRC (Read error: Connection reset by peer) [06:28] *** britmob has joined #archiveteam-ot [07:05] *** dhyan_nat has joined #archiveteam-ot [07:43] *** ShellyRol has quit IRC (Read error: Connection reset by peer) [07:45] *** ShellyRol has joined #archiveteam-ot [07:45] *** qw3rty_ has joined #archiveteam-ot [07:46] *** lunik1 has quit IRC (Read error: Connection reset by peer) [07:46] *** qw3rty has joined #archiveteam-ot [07:47] *** Flashfire has quit IRC (Read error: Connection reset by peer) [07:47] *** Dallas has quit IRC (Read error: Connection reset by peer) [07:47] *** marked1 has quit IRC (Read error: Connection reset by peer) [07:47] *** benjinsmi has joined #archiveteam-ot [07:48] *** qw3rty__ has quit IRC (Ping timeout: 276 seconds) [07:48] *** OrIdow6 has quit IRC (Ping timeout: 276 seconds) [07:48] *** atphoenix has quit IRC (Ping timeout: 276 seconds) [07:48] *** benjins has quit IRC (Ping timeout: 276 seconds) [07:49] *** atphoenix has joined #archiveteam-ot [07:50] *** qw3rty_ has quit IRC (Ping timeout: 276 seconds) [07:50] *** marked1 has joined #archiveteam-ot [07:52] *** marked10 has joined #archiveteam-ot [07:59] *** marked1 has quit IRC (Ping timeout: 492 seconds) [08:00] *** marked101 has joined #archiveteam-ot [08:07] *** marked10 has quit IRC (Ping timeout: 492 seconds) [08:07] *** marked101 is now known as marked10 [08:08] *** marked108 has joined #archiveteam-ot [08:09] *** marked108 has quit IRC (Client Quit) [08:10] *** marked104 has joined #archiveteam-ot [08:11] *** marked104 has quit IRC (Client Quit) [08:12] *** marked105 has joined #archiveteam-ot [08:13] *** marked105 has quit IRC (Client Quit) [08:14] *** marked103 has joined #archiveteam-ot [08:15] *** marked103 has quit IRC (Client Quit) [08:16] *** marked10 has quit IRC (Ping timeout: 492 seconds) [08:33] *** OrIdow6 has joined #archiveteam-ot [09:03] *** OrIdow6 has quit IRC (Ping timeout: 276 seconds) [09:35] *** OrIdow6 has joined #archiveteam-ot [11:09] *** OrIdow6 has quit IRC (Ping timeout: 276 seconds) [11:29] *** BlueMax has quit IRC (Read error: Connection reset by peer) [11:41] *** OrIdow6 has joined #archiveteam-ot [12:28] *** LowLevelM has joined #archiveteam-ot [12:49] *** X-Scale` has joined #archiveteam-ot [12:56] *** X-Scale has quit IRC (Ping timeout: 610 seconds) [12:56] *** X-Scale` is now known as X-Scale [13:01] *** X-Scale` has joined #archiveteam-ot [13:07] *** X-Scale has quit IRC (Ping timeout: 610 seconds) [13:07] *** X-Scale` is now known as X-Scale [13:48] *** mystonlin has joined #archiveteam-ot [13:52] Thanks very much @nyany, it would be great if you could replace the citation links at the bottom of the page which currently both point to example.com for both, with the relevant forum posts: https://mystonline.com/forums/viewtopic.php?f=1&t=29240 and https://forums.cyan.com/viewtopic.php?f=7&t=1608 Thanks again! [13:54] *** mystonlin has quit IRC (Remote host closed the connection) [15:32] *** igloo25 has joined #archiveteam-ot [15:50] JAA: That was exactly it. Seems to work fine now. [15:50] in RE: the reference links [15:51] atphoenix: btw, the youtube captcha is exactly the same as google search captcha, even the google domain [16:33] *** britmob has quit IRC (Read error: Connection reset by peer) [16:33] *** marked10 has joined #archiveteam-ot [16:35] *** britmob has joined #archiveteam-ot [16:59] *** MaximeleG has joined #archiveteam-ot [17:05] *** MaximeleG has quit IRC (Quit: MaximeleG) [17:07] *** MaximeleG has joined #archiveteam-ot [17:07] *** thuban3 has quit IRC (Read error: Connection reset by peer) [17:09] *** thuban3 has joined #archiveteam-ot [17:22] *** DogsRNice has joined #archiveteam-ot [17:24] *** MaximeleG has quit IRC (Quit: MaximeleG) [17:30] *** godane1 has joined #archiveteam-ot [17:31] *** godane has quit IRC (Read error: Connection reset by peer) [17:35] *** thuban4 has joined #archiveteam-ot [17:41] *** thuban3 has quit IRC (Read error: Operation timed out) [17:50] *** Atom__ has quit IRC (Quit: Atom__) [17:54] Is anyone here familiar with perl? [17:55] I wanted to try out https://metacpan.org/pod/release/BSUGARS/IPC-Shareable-0.60/lib/IPC/Shareable.pm , it looks like it allows you to store a variable and reference it within any shell process [17:56] I'd love to use that in a bash script I'm working on, but I have absolutely no clue on how to use it. I've read through the page but perl is so foreign to me I don't know what I'm looking at [17:58] *** thuban has joined #archiveteam-ot [18:01] *** thuban4 has quit IRC (Ping timeout: 276 seconds) [18:29] *** marked108 has joined #archiveteam-ot [18:29] *** asdf01012 has joined #archiveteam-ot [18:30] *** marked10 has quit IRC (Read error: Operation timed out) [18:30] *** marked108 is now known as marked10 [18:32] *** asdf0101 has quit IRC (Read error: Operation timed out) [18:32] *** asdf01012 is now known as asdf0101 [18:51] systwi : I know perl. I'm not sure you want to use it from bash. Try to at least find a bash native way first. [18:51] *** marked10 is now known as marked1 [18:56] I sense an XY problem. [19:08] *** superkuh has quit IRC (Quit: the neuronal action potential is an electrical manipulation of reversible abrupt phase changes in the lipid bilaye) [19:40] *** qw3rty has quit IRC (Read error: Operation timed out) [19:59] #archiveteam needs ODAH!!!! [20:01] from hackint at -ot: The Best Ever Put-Downs From House Speaker John Berco https://www.youtube.com/watch?v=kAM-YW-6vdU <-- ODAH! ODAH! (Order! Order!) keeping #archiveteam on topic [20:02] *** superkuh has joined #archiveteam-ot [20:08] maybe we could have a talking JohnBercoBot (JBB) in #archiveteam. Yells ODAH! after (some criteria). Step 1 is clear (create bot). Step 2 (the criteria) is nebulous. Step 3 Profit. (southpark gnomes references) [20:09] true. systwi: perl is rarely the best choice. what was the original task you're trying to get done, what did you try, and where did you get stuck? [20:19] marked1: Thanks for the input. In my bash script I want it to make sure it's the only instance running, and if a second instance is running, wait until the first one is finished. I read online a lot of people suggest using temporary files, however there is the chance that a sudden power outage could shut down the computer improperly. Then, when it's booted back up the temporary files are still there. A new run of the sc [20:19] ript may see the temp file still there and think another iteration is running, when it's not [20:20] systwi: the usual method in this case is to apply `pgrep` [20:20] Yes, but this is a bash script. As far as I know the process will only be listed as "bash" [20:21] Which could easily get confused if I have other Terminal windows open doing other things, which is very common [20:21] Try flock https://www.systutorials.com/docs/linux/man/1-flock/ [20:22] sorry that site is full of 3rd party js, it's just the man page for flock which you likely have installed already though [20:22] Oh yeah I installed flock a long time ago (I'm using macOS so I had to use brew) [20:23] I'm reading the man page right now [20:23] what is the task you were trying to solve though, bash might not be a good language for it [20:23] *** qw3rty has joined #archiveteam-ot [20:23] The only thing is, though, that still means I have to rely on files and not something in memory [20:23] like a variable [20:24] I am writing a tool to archive everything for a specified YT channel. That's the brief summary, I have been working on this tool for about 7 months now, I reeeally prefer not to switch languages at this point lol [20:25] Everything has gone perfect so far [20:26] *** dhyan_nat has quit IRC (Read error: Operation timed out) [20:26] *** jodizzle has quit IRC (Read error: Operation timed out) [20:26] *** Frogging has quit IRC (Read error: Operation timed out) [20:26] *** Frogging has joined #archiveteam-ot [20:26] *** girst has quit IRC (Read error: Operation timed out) [20:26] *** jodizzle has joined #archiveteam-ot [20:26] *** dxrt_ has quit IRC (Read error: Operation timed out) [20:26] *** Craigle has quit IRC (Read error: Operation timed out) [20:27] *** systwiALT has joined #archiveteam-ot [20:27] Sorry about that, connection timeout, hope I didn't miss anything [20:27] *** paul2520 has quit IRC (Read error: Operation timed out) [20:28] *** Auctus has quit IRC (Read error: Operation timed out) [20:28] *** Auctus has joined #archiveteam-ot [20:28] *** Raccoon` has joined #archiveteam-ot [20:28] *** asdf0101 has quit IRC (Read error: Operation timed out) [20:28] put the lockfile in /tmp and it should achieve what you specify [20:29] *** LowLevelM has quit IRC (Read error: Operation timed out) [20:29] *** atphoeni1 has joined #archiveteam-ot [20:29] *** luckcolor has quit IRC (Read error: Operation timed out) [20:29] That's definitely a good idea, however what about the off chance another file with the same name already exists? [20:29] *** keith20 has quit IRC (Read error: Operation timed out) [20:29] *** luckcolor has joined #archiveteam-ot [20:30] That's why I thought having something in memory would prevent any issues [20:30] *** girst has joined #archiveteam-ot [20:30] There's also the possibility of the user having a full disk [20:30] I'm just trying to cover all bases [20:30] *** atphoenix has quit IRC (Read error: Operation timed out) [20:30] *** atphoeni1 is now known as atphoenix [20:31] @systwi to solve your temporary problem you can put the files in a RAM-disk (/dev/shm or /tmp depending on configuration). This will be lost upon power-off solving your problem. [20:31] if the disk is full you're not gonna download any YouTube videos anyway [20:31] trying to invent something nobody has thought of before is going to cause more bugs than following what's considered normal practice [20:31] SootBectr beat me to it. I was scrolled up. [20:32] *** Raccoon has quit IRC (Read error: Operation timed out) [20:32] *** Raccoon` is now known as Raccoon [20:32] The idea of a RAM disk is a good idea, idk if macOS's /tmp folder _is_ a ram disk in disguise. I can't reboot to check, too much stuff open atm lol. [20:32] Also to ensure a unique file name first create a folder for your program and append a unix timestamp to the end of the filename. [20:33] nicolas17: True, but the user could download it to a drive with free space [20:33] *** Tenebrae has quit IRC (Ping timeout: 864 seconds) [20:33] I was going to use a uuid [20:33] *** systwi has quit IRC (Read error: Operation timed out) [20:33] But I can also append a unix timestamp to that :) [20:33] Also have you tried youtube-dl? It can download an entire channel out of the box? [20:33] systwiALT: if you really want to avoid lockfiles, you can invoke script.sh as /bin/bash script.sh (instead of ./script.sh) and use `pgrep -f`. but this will have a similar problem in that a the process name could theoretically exist; at some point you have to trust your system. [20:34] mr_archiv: Yes, youtube-dl is a critical component, but I am logging the history of channels as well. One grab in 2016 could be identical to a grab in 2019, but the user could have changed their channel banner. My script will track all of that [20:34] s/a the/a similar/ [20:35] Got it [20:35] I also lost my EFnet connection for 5 minutes. For others thinking they have gaps, see https://archive.fart.website/bin/irclogger_log/archiveteam-ot?date=2020-01-26,Sun [20:35] To ensure a unique filename, use mktemp [20:36] Thanks for the tips everyone [20:36] Didn't know about mktemp too, nice [20:36] *** kiska has quit IRC (Ping timeout: 622 seconds) [20:36] I guess to be absolutely safe, I can; 1, check the disk space is > enough to make a temp file [20:37] 2, create the temp file with mktemp and flock it [20:37] Then any new script iterations will be on hold until the file is freed and removed [20:38] Now, regarding perl, I can't remember exactly what it was in my script, but I did use it (something like `perl -c "code here"`) and it worked perfectly [20:38] *** Wingy has quit IRC (Read error: Operation timed out) [20:38] That's why I thought maybe doing something similar with a non-tempfile-reliant solution would work [20:39] er, isn't the point of mktemp the randomization, i.e. that successive invocations will _not_ clobber each other? [20:40] each mktemp will return a different name, yes [20:40] *** MrRadar2 has quit IRC (Read error: Operation timed out) [20:40] I just also think of the possibility of the user unable to write to /tmp (would probably cause WAY more issues though) [20:41] It's probably me just being too paranoid [20:49] atphoenix: no bots that talk in #archiveteam [20:50] where did a bot talk? [20:50] no bot talked. I was trying to make a humorous comment. [21:00] *** godane1 has quit IRC (Ping timeout: 255 seconds) [21:06] *** Tenebrae has joined #archiveteam-ot [21:07] *** dhyan_nat has joined #archiveteam-ot [21:07] *** asdf0101 has joined #archiveteam-ot [21:07] *** LowLevelM has joined #archiveteam-ot [21:07] *** dxrt_ has joined #archiveteam-ot [21:08] *** keith20 has joined #archiveteam-ot [21:08] *** MrRadar2 has joined #archiveteam-ot [21:08] *** svchfoo3 sets mode: +o dxrt_ [21:08] *** svchfoo1 sets mode: +o dxrt_ [21:09] *** Wingy has joined #archiveteam-ot [21:14] *** godane has joined #archiveteam-ot [21:15] *** atphoenix has quit IRC (Read error: Connection reset by peer) [21:16] *** Frogging has quit IRC (Read error: Connection reset by peer) [21:16] *** atphoenix has joined #archiveteam-ot [21:17] *** thuban has quit IRC (Ping timeout: 276 seconds) [21:17] *** paul2520 has joined #archiveteam-ot [21:17] *** thuban has joined #archiveteam-ot [21:18] *** Frogging has joined #archiveteam-ot [21:18] *** qw3rty_ has joined #archiveteam-ot [21:18] *** qw3rty has quit IRC (Ping timeout: 276 seconds) [21:23] *** Raccoon has quit IRC (Ping timeout: 276 seconds) [21:24] *** Raccoon has joined #archiveteam-ot [21:24] *** OrIdow6 has quit IRC (Ping timeout: 276 seconds) [21:26] *** qw3rty_ has quit IRC (Read error: Connection reset by peer) [21:27] *** qw3rty has joined #archiveteam-ot [21:39] *** OrIdow6 has joined #archiveteam-ot [21:46] *** thuban1 has joined #archiveteam-ot [21:47] *** Ctrl has joined #archiveteam-ot [21:51] *** thuban has quit IRC (Read error: Operation timed out) [21:55] *** kiska has joined #archiveteam-ot [21:56] *** svchfoo3 sets mode: +o kiska [21:56] *** svchfoo1 sets mode: +o kiska [21:59] *** BlueMax has joined #archiveteam-ot [22:06] *** Craigle has joined #archiveteam-ot [22:07] *** dhyan_nat has quit IRC (Read error: Operation timed out) [22:17] *** NickN00b has joined #archiveteam-ot [22:50] *** antomati_ has joined #archiveteam-ot [22:50] *** qw3rty_ has joined #archiveteam-ot [22:51] *** ranma_ has joined #archiveteam-ot [22:53] *** benjins has joined #archiveteam-ot [22:54] *** yawkat` has joined #archiveteam-ot [22:54] *** Fionera_ has joined #archiveteam-ot [22:54] *** sknebel_ has joined #archiveteam-ot [22:55] *** qw3rty has quit IRC (irc.efnet.nl efnet.deic.eu) [22:55] *** britmob has quit IRC (irc.efnet.nl efnet.deic.eu) [22:55] *** X-Scale has quit IRC (irc.efnet.nl efnet.deic.eu) [22:55] *** benjinsmi has quit IRC (irc.efnet.nl efnet.deic.eu) [22:55] *** ShellyRol has quit IRC (irc.efnet.nl efnet.deic.eu) [22:55] *** dashcloud has quit IRC (irc.efnet.nl efnet.deic.eu) [22:55] *** ranma has quit IRC (irc.efnet.nl efnet.deic.eu) [22:55] *** sknebel has quit IRC (irc.efnet.nl efnet.deic.eu) [22:55] *** Kenshin has quit IRC (irc.efnet.nl efnet.deic.eu) [22:55] *** Polylith has quit IRC (irc.efnet.nl efnet.deic.eu) [22:55] *** Fionera has quit IRC (irc.efnet.nl efnet.deic.eu) [22:55] *** yawkat has quit IRC (irc.efnet.nl efnet.deic.eu) [22:55] *** chfoo has quit IRC (irc.efnet.nl efnet.deic.eu) [22:55] *** betamax has quit IRC (irc.efnet.nl efnet.deic.eu) [22:55] *** antomatic has quit IRC (irc.efnet.nl efnet.deic.eu) [22:56] *** betamax_ has joined #archiveteam-ot [22:56] *** chfoo_ has joined #archiveteam-ot [22:56] *** britmob_ has joined #archiveteam-ot [22:58] *** RKenshin has joined #archiveteam-ot [23:05] *** X-Scale` has joined #archiveteam-ot [23:07] *** Polylith_ has joined #archiveteam-ot [23:08] *** thuban1 has quit IRC (Read error: Connection reset by peer) [23:09] *** thuban1 has joined #archiveteam-ot [23:10] *** dashcloud has joined #archiveteam-ot [23:10] *** RKenshin is now known as Kenshin [23:12] *** ShellyRol has joined #archiveteam-ot [23:47] *** chfoo_ is now known as chfoo