Lord_Nigh: Considering the IA staffer in question was Jason, I think a more likely case than legal is "Jason is known to oppose robots.txt entirely, so ask him not to talk about IA's policies on it" hmm. could be. he may be pissed off about the current situation but his hands are tied who knows If some *other* IA staffer were to say "I'm actually not allowed to comment on the robots.txt thing", I'd be a lot more likely to point to legal stuff. true for all i know the entire "sudden change" in the way robots.txt is a bug but if so i have no idea who is in charge of fixing it I have no doubt Jason would prefer to see the Wayback Machine ignore robots.txt, but I am also certain it isn't his decision. And I know he pissed off various people when he talked about the proposed changes earlier, so it doesn't surprise me *at all* that he was asked not to talk about it further. pissed off various people *OUTSIDE IA* important clarification. I have no idea what anyone actually connected with IA felt about it. I'm also pretty amused/frustrated by MogMiner's comment: "given how open IA usually is, I'm just baffled why this is such a touchy subject with them" It's really *obvious* to me at least why this is a touchy subject! IA's fundamental way to deal with people upset about them distributing stuff is to say, "OK, sorry, we've stopped now. Bye!" Adjusting robots.txt functionality in Wayback (either way) affects that -- *of course* they are touchy about it! And the tactic (of saying, sorry-we've-stopped-now) works better the less attention is paid to it -- so minimizing what is said about things ... ... associated with the topic is basic sensible strategy. OK, rant over. tapedrive: http://database.savefanfiction.tk/ throws a 404 Sorry, the requested URL 'http://database.savefanfiction.tk/' caused an error:Not found: '/' Somebody2: also don't prevent from saving anyway and just not displaying gotta think of the long game :) jrwr: wait, what? I think I forgot the thread of this conversation. Yarg! Evil Robots.txt :) Somebody2: Having Wayback ignore Robots.txt, it could still store and not display I would suspect that is whats going on Ah, I see. I bet its just some temp bullshit going down I agree that is likely, yes. Somebody2: just got done setting up a Onion Enabled ArchiveBot jrwr: I saw -- good work! I feel like that is a ever growing gap I know there can be some crazy shit, but its still our culture another possibility is that they're trying different robots.txt behaviours to see how much people complain about them, perhaps davidar: that's an ... amusing ... possibility hahaha I love it I went on a watching spree of defcon and other con videos man I think ive watched over 3-4 hours of SketchCow talking now jrwr: only 4 hours? There's lots more... I know, just haven't hunted them down "yet" jrwr: http://ascii.textfiles.com/speaking jrwr: it's a list, not a specific one Im adding them all nods He is good fun Right now im watching 3C33 on a 6502 found in a cable box I don't know why, but i love the 6502, I've done a ton of arcade machine repair and love the layout it promotes I need to publish my work on a clip on SRAM tester and 6502 Debugger it uses a fuck ton of pogo pins, but allows me to sniff all the data off the 6502 as it comes in and out, and the rPi keeps up with it in a 6502 emulated CPU, this allows the rPi to in theory replace the 6502 on board according to https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/#comment-359347 "ia_archiver" isn't even wayback's useragent (anymore?) davidar: interesting arkiver: I'm still seeing the infinite 206s for IMzy It's not on every item, but I've got a big backlog of items on my VPSes with them MrRadar: isn't that expected? 206 just means there's more to download But fot literally 3000+ requests in a row? 3514=206 https://www.imzy.com/api/accounts/profiles/mcnulty?check=true For the same URL yeah, i got a lot of those too, i guess i assumed it was fetching some more data or something the page is empty, maybe it's imzys way of saying come back later? dunno I get different sized responses when refreshing in Chrome MrRadar: i'm beginning to suspect that you're right, something is very strange with the imzy archiver p/win 2 erji0jwq9r1r0rjaodsjfiq dsaffdiqi023 rong. Verizon closed on their acquisition of Yahoo: https://techcrunch.com/2017/06/13/verizon-closes-4-5b-acquisition-of-yahoo-marissa-mayer-resigns-memo/ Kaboom I have a feeling its going to die now its going to get renamed Veriyoo Interesting link from /r/DataHoarder: https://it.reddit.com/r/DataHoarder/comments/6gyspx/2_tib_of_old_scene_games_finally_released/ nice, thanks Grab it, Catalog it, and store it Someday, some one will want it thats how I see it even if they can't use it for the next 50 years if we dont save it, no one else will, save it for the future when 100 years from now we can show them how all this went down \