#archiveteam-bs 2019-07-12,Fri

↑back Search

Time Nickname Message
00:00 πŸ”— killsushi has joined #archiveteam-bs
00:05 πŸ”— LowLevelM has joined #archiveteam-bs
00:06 πŸ”— astrid has joined #archiveteam-bs
00:06 πŸ”— Fusl sets mode: +o astrid
00:07 πŸ”— LowLevelM Ok Fusl, How do you get 5Tbps? I assume that is spread across many servers. right?
00:07 πŸ”— Fusl 500 servers each 10gbit
00:07 πŸ”— LowLevelM 500 servers. What?
00:08 πŸ”— LowLevelM That must be soo expensive
00:08 πŸ”— Fusl Β―\_(ツ)_/Β―
00:09 πŸ”— JAA -ot this?
00:09 πŸ”— Fusl no more comments need to be made so no
00:13 πŸ”— systwi has joined #archiveteam-bs
00:20 πŸ”— arkiver Fusl: :O
00:21 πŸ”— arkiver that is niiiiice
00:54 πŸ”— qwebirc20 has quit IRC (Ping timeout: 261 seconds)
00:54 πŸ”— LowLevelM has quit IRC (Ping timeout: 261 seconds)
01:14 πŸ”— LowLevelM has joined #archiveteam-bs
01:19 πŸ”— LowLevelM has quit IRC (Ping timeout: 260 seconds)
01:38 πŸ”— LowLevelM has joined #archiveteam-bs
01:38 πŸ”— HashbangI has quit IRC (Read error: Connection reset by peer)
01:46 πŸ”— HashbangI has joined #archiveteam-bs
02:31 πŸ”— BlueMax has joined #archiveteam-bs
03:09 πŸ”— qw3rty118 has joined #archiveteam-bs
03:15 πŸ”— qw3rty117 has quit IRC (Read error: Operation timed out)
03:48 πŸ”— odemgi_ has joined #archiveteam-bs
03:49 πŸ”— odemg has quit IRC (Read error: Operation timed out)
03:50 πŸ”— odemgi has quit IRC (Read error: Operation timed out)
04:04 πŸ”— odemg has joined #archiveteam-bs
05:02 πŸ”— killsushi has quit IRC (Read error: Operation timed out)
05:42 πŸ”— m007a83_ is now known as m007a83
06:19 πŸ”— LowLevelM has quit IRC (Ping timeout: 260 seconds)
06:58 πŸ”— schbirid has joined #archiveteam-bs
07:11 πŸ”— Atom has quit IRC (Ping timeout: 252 seconds)
07:12 πŸ”— Atom has joined #archiveteam-bs
07:53 πŸ”— SketchCow I'm now running a script to see how many of the mirrored youtube videos are missing from youtube.
07:59 πŸ”— Fusl SketchCow: on a related note, have you noticed that youtube is now aggressively blocking/rate-limiting youtube-dl and similar downloaders such that mass mirroring of youtube channels is not easily possible anymore?
08:01 πŸ”— Fusl https://torrentfreak.com/youtube-blocks-popular-mp3-stream-ripping-sites-190710/
08:02 πŸ”— Flashfire it wasnt as big a problem blocking the sites but now blocking the software itself
08:27 πŸ”— MillerBOS has quit IRC (Read error: Connection reset by peer)
08:27 πŸ”— pikami_ has quit IRC (Write error: Broken pipe)
08:27 πŸ”— odemgi_ has quit IRC (Write error: Broken pipe)
08:27 πŸ”— thejsa has quit IRC (Write error: Broken pipe)
08:27 πŸ”— dashcloud has quit IRC (Write error: Broken pipe)
08:27 πŸ”— m007a83_ has joined #archiveteam-bs
08:27 πŸ”— benjinss has joined #archiveteam-bs
08:27 πŸ”— odemgi_ has joined #archiveteam-bs
08:27 πŸ”— benjinss has quit IRC (Read error: Connection reset by peer)
08:27 πŸ”— MillerBOS has joined #archiveteam-bs
08:28 πŸ”— thejsa has joined #archiveteam-bs
08:28 πŸ”— dashcloud has joined #archiveteam-bs
08:28 πŸ”— pikami has joined #archiveteam-bs
08:29 πŸ”— benjinss has joined #archiveteam-bs
08:33 πŸ”— stapler11 has quit IRC (Read error: Operation timed out)
08:33 πŸ”— benjinsmi has quit IRC (Ping timeout: 604 seconds)
08:33 πŸ”— m007a83 has quit IRC (Read error: Operation timed out)
08:34 πŸ”— stapler11 has joined #archiveteam-bs
08:40 πŸ”— Igloo has quit IRC (Read error: Operation timed out)
08:40 πŸ”— Igloo has joined #archiveteam-bs
08:44 πŸ”— LeG0ax has joined #archiveteam-bs
08:45 πŸ”— RichardG has quit IRC (Read error: Operation timed out)
08:45 πŸ”— RichardG has joined #archiveteam-bs
08:46 πŸ”— Ing3b0rg has quit IRC (Ping timeout: 506 seconds)
08:46 πŸ”— LeG0ax is now known as Ing3b0rg
08:47 πŸ”— nyany has quit IRC (Read error: Operation timed out)
08:48 πŸ”— svchfoo3 has quit IRC (Ping timeout: 506 seconds)
08:49 πŸ”— eientei95 has quit IRC (Ping timeout: 506 seconds)
08:49 πŸ”— PurpleSym has quit IRC (Read error: Operation timed out)
08:49 πŸ”— purplebot has quit IRC (Read error: Operation timed out)
08:49 πŸ”— pikami has quit IRC (Ping timeout: 506 seconds)
08:50 πŸ”— pikami has joined #archiveteam-bs
08:50 πŸ”— PurpleSym has joined #archiveteam-bs
08:51 πŸ”— eientei95 has joined #archiveteam-bs
08:51 πŸ”— eientei95 has quit IRC (Handshake flooding)
08:53 πŸ”— h3ndr1k_ has joined #archiveteam-bs
08:53 πŸ”— eientei95 has joined #archiveteam-bs
08:53 πŸ”— eientei95 has quit IRC (Handshake flooding)
08:54 πŸ”— h3ndr1k has quit IRC (Ping timeout: 740 seconds)
08:56 πŸ”— eientei95 has joined #archiveteam-bs
09:00 πŸ”— h3ndr1k_ is now known as h3ndr1k
09:43 πŸ”— nyany has joined #archiveteam-bs
09:44 πŸ”— purplebot has joined #archiveteam-bs
09:44 πŸ”— svchfoo3 has joined #archiveteam-bs
09:44 πŸ”— Fusl sets mode: +o svchfoo3
10:10 πŸ”— betamax_ is now known as betamax
10:22 πŸ”— deevious has joined #archiveteam-bs
11:12 πŸ”— Raccoon has joined #archiveteam-bs
11:28 πŸ”— BlueMax has quit IRC (Read error: Connection reset by peer)
11:46 πŸ”— Fusl_ JAA: where do you want the nratv stuff uploaded?
12:39 πŸ”— Fusl_ fuzzy8021: you around?
12:45 πŸ”— Fusl_ arkiver: fyi, i'm pulling flickr out of jrwr's storage now and soon doing the others as well so if you have anything still running that pulls data together from there, now is a good time to kill all of that
12:46 πŸ”— JAA Fusl_: Ah, right, NRATV. So you have ~20k WARCs and ~20k video files, right?
12:46 πŸ”— Fusl_ 22188
12:47 πŸ”— JAA Probably best to coordinate this with IA.
12:47 πŸ”— JAA We'll want the video files as items I assume, with the appropriate metadata.
12:47 πŸ”— Fusl_ do we need JS for this or do you have contact with people at IA?
12:48 πŸ”— JAA Not sure about the WARCs, either as they are or megawarcs I guess.
12:48 πŸ”— Fusl they're currently not megawarced
12:48 πŸ”— JAA Jason's probably the guy for that. I haven't spoken with anyone about that.
12:48 πŸ”— Igloo What do you need from IA Fusl?
12:48 πŸ”— Igloo I can go poke the slack.
12:48 πŸ”— JAA We'll want an "NRATV" collection I think.
12:49 πŸ”— Fusl ideally we want two i guess, one for the videos and one for the raw warc files that contains the videos
12:50 πŸ”— JAA Yeah, "NRATV" for the videos and "NRATV WARCs" for the WARCs?
12:50 πŸ”— Mateon1 has quit IRC (Read error: Operation timed out)
12:50 πŸ”— Fusl whatever is fine for them
12:51 πŸ”— JAA It would be even better if we could throw video and WARC in one item, but that doesn't work I think due to the mediatype.
12:51 πŸ”— Mateon1 has joined #archiveteam-bs
12:51 πŸ”— JAA Will have to extract the metadata also. I'll look into that later.
12:59 πŸ”— Igloo If you don't get a response from JS or arkiver etc I can ping the slack when we know what we want.
13:26 πŸ”— fuzzy8021 sup Fusl_
13:29 πŸ”— Fusl fuzzy8021: 95.216.12.47 is yours, right?
13:29 πŸ”— fuzzy8021 yep
13:30 πŸ”— fuzzy8021 do you need it?
13:32 πŸ”— luckcolor has quit IRC (Ping timeout: 246 seconds)
13:34 πŸ”— Fusl if you dont need it anymore, i'd like to take over the server into my hetzner account so you dont have to pay for it anymore
13:37 πŸ”— fuzzy8021 sure why not. havent gotten around to using it yet
14:36 πŸ”— arkiver Fusl: I donΒ΄t have anything pulling from there
14:36 πŸ”— arkiver and thanks for working on it!
14:38 πŸ”— deevious has quit IRC (Quit: deevious)
15:18 πŸ”— luckcolor has joined #archiveteam-bs
15:22 πŸ”— SketchCow What up
15:26 πŸ”— JAA SketchCow: 22k NRATV videos, each has a video file and a WARC (containing the playlist and all video segments)
15:27 πŸ”— Verified_ has quit IRC (Ping timeout: 252 seconds)
15:27 πŸ”— JAA Metadata isn't ready yet, but I think I have it somewhere.
15:27 πŸ”— SketchCow OK... so we want to make a collection? OK.
15:27 πŸ”— SketchCow Isn't some stuff up
15:27 πŸ”— JAA Yeah
15:29 πŸ”— SketchCow archiveteam_nratv now exists
15:29 πŸ”— killsushi has joined #archiveteam-bs
15:29 πŸ”— SketchCow Is there a consistency of naming of what's already up I can use to shove them in?
15:45 πŸ”— JAA I don't think anything's uploaded yet. At least not from us.
15:46 πŸ”— SketchCow OK, so just upload them, I'll shove them into the collection when you're ready.
15:46 πŸ”— SketchCow Or someone can ping me with access requests
15:46 πŸ”— SketchCow But I set it up and gave it an NRATV bio and whee
15:46 πŸ”— JAA Fusl_: ^ (Or if you want me to do it, let me know.)
15:46 πŸ”— SketchCow So I tried an experiment that failed
15:46 πŸ”— SketchCow I want to take a Youtube iD and know if the video's gone or not.
15:47 πŸ”— SketchCow I can't find a consistent way to check.
15:47 πŸ”— SketchCow There MUST be something out there
15:57 πŸ”— Fusl SketchCow: `test 200 == $(curl -sfo/dev/null -w '%{http_code}' "http://www.youtube.com/oembed?url=http://www.youtube.com/watch?v=${ID}")`
15:58 πŸ”— SketchCow Damn, that's dense
15:58 πŸ”— SketchCow Is that bash?
15:58 πŸ”— Fusl aye
16:03 πŸ”— SketchCow What are the possible outputs
16:03 πŸ”— SketchCow Because for me it outputs blank
16:03 πŸ”— Fusl it will give an exit value of either 0 or 1
16:03 πŸ”— Fusl so you can use it within an if-condition
16:03 πŸ”— SketchCow Not here
16:03 πŸ”— astrid or follow it with && echo $?
16:04 πŸ”— Fusl ; echo $?
16:04 πŸ”— astrid er right
16:04 πŸ”— Fusl && echo $? would only print if it succeeds
16:04 πŸ”— SketchCow I don't want to seem ungrateful
16:04 πŸ”— astrid computers.
16:04 πŸ”— SketchCow But man, that's dense
16:05 πŸ”— SketchCow Also, the whole endeavor is getting right into my face how much absolute horseshit people upload to the archive
16:05 πŸ”— SketchCow Which is not a mood lightener
16:06 πŸ”— SketchCow Oh, 5,000 hours of thai television..... thank you
16:06 πŸ”— SketchCow Especially with the 100%, complete and utter lack of metadata
16:06 πŸ”— SketchCow The robots after I'm dead will thank you
16:07 πŸ”— SketchCow BOB=`test 200 == $(curl -sfo/dev/null -w '%{http_code}' "http://www.youtube.com/oembed?url=http://www.youtube.com/watch?v=${ID}");echo $?`;echo $BOB
16:08 πŸ”— Raccoon start a streaming service that requires viewers to fill out metadata for you
16:09 πŸ”— SketchCow for each in `ia search collection:archiveteam_youtube --itemlist`; do YT=`echo $each | sed 's/youtube-//g'`; FOF=`test 200 == $(curl -sfo/dev/null -w '%{http_code}' "http://www.youtube.com/oembed?url=http://www.youtube.com/watch?v=${ID}");echo $?`;echo "$FOF"; if [ "$FOF" = "1" ]; then echo "$YT exists."; else echo "Oh no.... $YT is gone gone gone!"; echo "$each" >> deads.txt; fi;
16:09 πŸ”— SketchCow done
16:09 πŸ”— SketchCow What could possibly go wrong
16:09 πŸ”— Igloo Be careful you don't get banned by YT
16:09 πŸ”— Igloo Not sure how they're testing that.
16:10 πŸ”— SketchCow Oh no
16:10 πŸ”— SketchCow banned by YT
16:10 πŸ”— SketchCow What will I do
16:10 πŸ”— SketchCow How will I spend that free time
16:10 πŸ”— Igloo It will break your script.
16:11 πŸ”— Igloo That's all I was saying.
16:11 πŸ”— Fusl for gods sake make JAA use mips for this! :P
16:11 πŸ”— SketchCow 0idOIGRrbHU exists.
16:11 πŸ”— SketchCow 1
16:11 πŸ”— SketchCow 0ikhVJCblnk exists.
16:11 πŸ”— SketchCow 1
16:11 πŸ”— SketchCow 0j6aV3YSue8 exists.
16:12 πŸ”— SketchCow I'm mostly interested in seeing how many of these are actually missing
16:12 πŸ”— SketchCow And how many are straight up mirrors
16:12 πŸ”— arkiver IΒ΄m putting my money on 0.8%
16:12 πŸ”— Fusl 0.3%
16:13 πŸ”— SketchCow I've only got to work from the de-indexed set
16:13 πŸ”— SketchCow NON-de-indexed
16:13 πŸ”— arkiver IΒ΄m putting my second money on Fusl being correct
16:13 πŸ”— Fusl arkiver: thats not how it works :P
16:13 πŸ”— arkiver :)
16:13 πŸ”— SketchCow You bid one dollar over
16:13 πŸ”— SketchCow And fuck them
16:13 πŸ”— SketchCow (That's how the Price is Right works)
16:14 πŸ”— Fusl im too young for this
16:17 πŸ”— SketchCow By the way - so far none are missing.
16:18 πŸ”— SketchCow I choose random youtube IDs to go make sure things are fine, and I have not been delighted at the video chosen to be mirrored.
16:18 πŸ”— SketchCow Which tells me they're not choosing. They're mirroring almost random things
16:19 πŸ”— arkiver mirroring whatever they find personally interesting
16:19 πŸ”— SketchCow No, I don't think so
16:19 πŸ”— SketchCow No, no.
16:19 πŸ”— arkiver although thereΒ΄s exceptions among those people
16:19 πŸ”— Igloo How many times have you been Rick Rolled?
16:19 πŸ”— SketchCow Not when you mirror 15,000 videos
16:19 πŸ”— SketchCow No, that's just high-spectrum grab-bag snowplowing through someone else's harddrives
16:20 πŸ”— arkiver yeah true
16:24 πŸ”— Raccoon has quit IRC (Ping timeout: 265 seconds)
16:25 πŸ”— SketchCow Yeah, so far, zero percent down.
16:25 πŸ”— SketchCow Waiting for my ban
16:25 πŸ”— SketchCow DO IT
16:25 πŸ”— SketchCow DOOOO IT
16:25 πŸ”— schbirid i say 1.5% are gone
16:41 πŸ”— SketchCow I say that before we're done, two will die, and one will be irrevocably changed
17:39 πŸ”— Verified_ has joined #archiveteam-bs
18:06 πŸ”— Ryz has joined #archiveteam-bs
18:14 πŸ”— m007a83_ is now known as m007a83
19:57 πŸ”— betamax speaking of YouTube archiving, is ivan still the one running the GDrive-based archiver that only uploads videos once they're taken down?
19:57 πŸ”— betamax or is that now someone else?
20:00 πŸ”— icedice has joined #archiveteam-bs
20:03 πŸ”— Igloo It was, but it's also been banned mostly.
20:05 πŸ”— betamax ah, shame
20:33 πŸ”— SketchCow With the caveat that we made this shit up on the spot, 0% of the URLs I had access to are not still in youtube.
20:47 πŸ”— ivan_ SketchCow: I found that after a few years, ~8% of my YouTube was gone from YouTube
20:47 πŸ”— ivan_ but I'm not a tubeupper
20:49 πŸ”— stapler11 has quit IRC (Leaving)
20:56 πŸ”— betamax Hypothetical question (asking here before I bother info@archive.org), anyone know if IAs system allows for items uploaded to one account to be transferred to another?
20:57 πŸ”— Smiley so can we use Google Compute credit?
21:00 πŸ”— SketchCow Tell me the circumstances this would happen
21:02 πŸ”— Smiley guy I know goes "I have Β£283 worth of Google Compute Dealie credit though, if anyone can think of a use for it?"
21:02 πŸ”— Smiley I'm not sure if we can use the warrior scripts on it, or something
21:03 πŸ”— hook54321 PurpleSym: Awhile ago you asked if the Circavie archives exist anywhere, did you ever find them?
21:06 πŸ”— Igloo Smiley: the outbound would be the issue
21:06 πŸ”— ivan_ I ran grab-site on GCE trial credit and got my servers and API project were removed with no warning
21:06 πŸ”— Igloo and I think he was reffering to the movement of items to anothe ruser
21:06 πŸ”— Smiley ivan_: dafaq :/
21:20 πŸ”— astrid betamax: yes it can be done
21:22 πŸ”— betamax good to know, thanks (don't want to waste time on impossible requests)
21:29 πŸ”— SketchCow betamax: As I wrote: Tell me the circumstances this would happen
21:33 πŸ”— betamax oh, sorry, thought you meant someone else
21:33 πŸ”— SketchCow In general, we entertain all requests but it should be for a good reason.
21:33 πŸ”— betamax basically I started writing scripts to mirror UK council webcasts (which are deleted after a set time) to IA, and initially used my personal IA account
21:33 πŸ”— SketchCow If someone's trying to put one over, we'll suss it out.
21:34 πŸ”— SketchCow But if you're able to prove you can log into both accounts, the effort is trivial.
21:34 πŸ”— betamax now I realise there's so many that it would be better to have a dedicated account as all my other items on that account are getting buried
21:34 πŸ”— SketchCow Yes.
21:34 πŸ”— SketchCow What we would do is:
21:34 πŸ”— betamax (this is currently hypothetical as I'm in the midst of re-writing the script and haven't made the second dedicated account yet)
21:35 πŸ”— SketchCow - Mail your old account's mailing address saying "You requested we do this. Is this you?"
21:35 πŸ”— SketchCow And you go yes.
21:36 πŸ”— betamax great. It won't be for a few weeks (finishing scripts, updating VM to debian 10, etc...) but knowing it is possible is a big help
22:01 πŸ”— LowLevelM has joined #archiveteam-bs
22:06 πŸ”— LowLevelM has quit IRC (Ping timeout: 260 seconds)
22:10 πŸ”— LowLevelM has joined #archiveteam-bs
22:22 πŸ”— LowLevelM has quit IRC (Ping timeout: 260 seconds)
22:23 πŸ”— SketchCow It is.
22:23 πŸ”— SketchCow You can come to me.
22:59 πŸ”— BlueMax has joined #archiveteam-bs
23:01 πŸ”— LowLevelM has joined #archiveteam-bs
23:04 πŸ”— schbirid has quit IRC (Remote host closed the connection)
23:35 πŸ”— yano_ is now known as yano
23:57 πŸ”— odemgi_ SketchCow, get this shit.... people think I'm you/you're me and that it's you that runs the-eye
irclogger-viewer