Is it strange that I find it fun to run a Rsync target JRWR: you might be a datahoarder :P LOOK AT THESE GRAPHS http://jrwr.io:19999 lol I love graphs, I look at my munin ones all the time for no reason :p netdata has awesome live graphs JRWR: you only have 1 GB of RAM? On that box its 16GB oh, I didn't even look at "cached" Its handling the Pixiv project right now is pixiv going down soon or something? I haven't been paying attention lol I turned on my nas after it being off for a few weeks, the rootfs is hosed john@oblivion:~$ htop -bash: /usr/bin/htop: Input/output error are SSDs really that bad without power Odd0002: chat.pixiv is ok Are there any media outlets that have said who the attackers in London are, or anything about them? Frogging: If they're worn-out, maybe? A healthy SSD should definitely be able to go a few weeks without power yeah I don't know yet if the SSD is broken or just the filesystem I don't trust it anymore either way though Yeah. At the very least I'd do a secure erase on it to reset its internal state MrRadar: Man we are blazing now we are up 3x the speed now Yeah, FOS has lots of storage but it taps out fairly quickly on IOPS All I got is 8TB I was thinking about that cross uploading to FOS to make sure in case of my server failing there is a backup but I would do it in big chunks wait what, SSDs die if they're not on? They *can* but any decent drive should be able to go for a year or more unpowered Yes Bit Fade Their flash cells leak electrons Even USB Drives do And SD/CF/whatever flash cards Spinning Rust will Decay as well but it takes MUCH longer how long does it take for SSD's? I know I have a 20 GB HDD that came with win98 that still works when was the last time it was powered on it's still on oh, the HDD? a few weeks ago I found an article on SSD unpowered data retention: http://www.anandtech.com/show/9248/the-truth-about-ssd-data-retention Includes this graph, showing how many weeks a drive is expected to retain data based on the temperature at which the data was written and the temperature the drive is stored: http://images.anandtech.com/doci/9248/3_575px.PNG Gotta keep them powered onj Mostly every type of storage will decay huh, guess I need to heat up my SSD while it's on lol lol, my rsync MOTD shows up on the warriors maybe attach a peltier device to my SSD that heats one side when the PC is on and cools it when its off if that data is correct then it might lengthen the data's lifetime if I heat it to 55C and cool it to 25 when its off then I could get 8x the data lifetime! Reading the article, that chart is for a drive at the end of its lifespan. Newer drives should retain data much longer I know, but now I know what to do with my SSD after it's near the end of its lifespan heat it when I power it on then put it in the freezer afterwards Its like playing a incermental game, watching the numbers go up and up Speaking of rsync, can we have seesaw detect when rsync failed because the files don't exist and just fail that job? Right now I'm going to have to cancel 5 jobs because the 6th is stuck in an infinite upload failure loop lol What project? is it savepixiv? It's for SPUF But I had to do the same for some a pixiv pipeline earlier today MrRadar: it should never try to upload nonexistent files in the first place? Ya That should be the pipline's fault I agree with that part too, but since it seems like it happens every so often it should probably still be handled Ya, overall if a job keeps failing outright the job manager should just nuke it and send it back MrRadar: have you filed a bug about this occurring? No, but that's a good idea rsync trying to upload nonexistent files sounds indicative of a bigger, more serious issue :) it's an invalid state that should never occur Looks like there's already a bug for the missing data issue: https://github.com/ArchiveTeam/seesaw-kit/issues/48 MrRadar: hold on, that's failing on a nonexistent *directory* not nonexistent *files* Yeah, sorry for not being precise, that's the exact issue I'm seeing LOL, I even commented on the issue at the very end Almost exactly 1 year ago I would love to become a secondary rsync ingress for AT have it auto sync the uploaded data to FOS and clear out the old when I have confirmed data upload MrRadar: hm :/ do it in bulk transfers and such since FOS iops are kind of low afk 1 hour joepie91, arkiver: Just skimmed through the logs, but that “garbage” looks alot like HTTP chunked transfer encoding to me. rsync error: errors selecting input/output files, dirs (code 3) at flist.c(2118) [sender=3.1.1] Process RsyncUpload returned exit code 3 for Item threads:2755750-2755759 ^ so this what you were discussing 12 hours ago just happened to me too PurpleSym: There are allegedly some instances where the garbage is space-delimited instead of newline-delimited. guys, is 6 warriors going to get me IP banned from anything besides yahoo and that other project? Got an example URL, timmc? I don't, just reporting what I saw in chat. Hello, I know you guys aren't archive.org, but do you know if there's a channel for it? Tried ##archive on freenode but it's invite-only. PurpleSym: There was this *very* interesting report joepie91 generated for (I think) just one URL, and by gosh it does look like garbled chunked transfer-encoding: http://sprunge.us/RjWi Would be interesting to look at the actual WARC. There are overlapping context chunks that have the right byte counts. (And look, there's a zero at the end.) fio67: There's #internetarchive here on EFNet but I'm not sure if it's "official" MrRadar: thanks, I'll take a look PurpleSym: I think https://archive.org/download/archiveteam_portalgraphics_20160727140857/portalgraphics_20160727140857.megawarc.warc.gz and https://web.archive.org/web/20160724001629/http://www.portalgraphics.net/pg/illust/?image_id=10575 That URL is not listed in the CDX as far as I see. PurpleSym: timmc: transfer chunk encoding sizes are decimal though, not hexadecimal? that having been said it does seem to generally match up in terms of size Nope, hex: https://tools.ietf.org/html/rfc2616#section-3.6.1 huh. joepie91: Those hex chunks that are space-delimited... is that only in the version displayed on web.archive.org, or also in the WARC? no idea, haven't checked the source. I think it's easier for arkiver to check that All the DATA, Give me maor! So as a just in case it happens when I get to 75% full on my rsync ingress server, what do add more HDDs :p its a OVH Box JRWR: also, you probably should be in #DataHoarder on Freenode :P I am already :3 anyway, that wasn't totally serious advice oh, you are? well once I get my server back not under this nick though? ah ya my 190TB Plex Archive is very nice I used my main server as a Weechat instance since I blanked it for this project I dont have that setup atm since I have 1Gbps up going to waste, I though about syncing to FOS but I would want to talk to SketchCow before I did, make sure everything was secure I could do bulk uploads to it, then purge the old jobs that I have confrimed on FOS but I've got another 6TB to go, so we have tons of time if its even needed at all 190TB on ovh? what do you pay? im guessing plex cloud synced with a google drive thatd be a hefty chunk of change :D I got one of the "Unlimted" google drive accounts so thats where it is UnEncrypted so it can be deduped ah nice The server I am using now for rsync ingress was my general server (had plex, website, other crap) I wiped it and raid0 it for this, and man I'm impressed my main is an EG-16 they recently lowered the prices on them but im locked in at my original price :( Thats what I'm using im at 78$ same 79 it's like 74 now lol I really want to find another one that is cheaper with the same stats dunno if youll be able to unmetered BW is the bomb everybody wants unmetered, but noone wants to pay for it :P ^^ i like cheap hetzner auction servers JRWR: I'll handle the uploads to IA or FOS 6tb for ~30€ but limited bw "unlimited" bw on that one nobody will beat OVH for unmetered ^ that's pretty much certain anybody who does will almost certainly go bankrupt in under a year that's why they have my service hehe (and they're usually just OVH resellers anyway) unlimited as in over 20TB = your 1gbit goes 10mbit (20TB outgoing) so it's really not that bad for a hetzner server like, probably the only reason OVH can get away with unmetered with a relatively good network is that they have a massive network of their own none of the other big names in budget servers have that tier 1 like, yeah and none of the new players will have it either, at least not for a while arkiver: still got the logins I gave you joepie91: theyre expanding rapidly too hence why it's pretty much certain that nobody will beat OVH without giving up on quality for unmetered stuff :P voidsta: yeah been following the ceo on twitter new dc here new dc there everyone gets a dc lol hehe, was about to make that joke :D JRWR: yes Cool also the scaleway arm64 instances are NICE they are faster then the x86/arm ones OVH is really the only name in dedicated servers without the fuss Atleast state side most of the other places are just meh agree Is this page broken for anyone else? https://archive.org/search.php?query=collection%3Aarchivebot&sort=-publicdate The HTML just stops somewhere in the navigation links. works for me Hm, weird. it has magic scrolling going on when you get to the bottom I know that, but I only get a partial navigation bar, no content at all.
  • BLOG