#archiveteam-bs 2017-01-26,Thu

↑back Search

Time Nickname Message
00:20 🔗 zerkalo has joined #archiveteam-bs
00:27 🔗 hook54321 pizzaiolo: Checked, it's already there.
00:28 🔗 hook54321 We might want to consider doing it soon though
00:29 🔗 nickname_ has quit IRC (Read error: Operation timed out)
00:32 🔗 espes__ has joined #archiveteam-bs
00:47 🔗 odemg has joined #archiveteam-bs
00:54 🔗 odemg has quit IRC (Remote host closed the connection)
01:02 🔗 godane has left
01:21 🔗 odemg has joined #archiveteam-bs
01:44 🔗 BlueMaxim has joined #archiveteam-bs
01:49 🔗 Darkstar has quit IRC (Ping timeout: 506 seconds)
01:58 🔗 icedice has quit IRC (Quit: Leaving)
02:09 🔗 kristian_ has quit IRC (Quit: Leaving)
02:16 🔗 vitzli has joined #archiveteam-bs
02:18 🔗 yan has quit IRC (Read error: Operation timed out)
02:28 🔗 nickname_ has joined #archiveteam-bs
02:29 🔗 Darkstar has joined #archiveteam-bs
02:40 🔗 schbirid2 has joined #archiveteam-bs
02:43 🔗 username1 has quit IRC (Read error: Operation timed out)
03:01 🔗 zhongfu has quit IRC (Ping timeout: 260 seconds)
03:04 🔗 pizzaiolo has left
03:08 🔗 godane has joined #archiveteam-bs
03:08 🔗 godane looks like ftp://aftp.cmdl.noaa.gov/ is gone
03:09 🔗 alembic welp
03:09 🔗 alembic glad we did that one first
03:12 🔗 Frogging nice
03:20 🔗 vitzli has quit IRC (Quit: Leaving)
03:27 🔗 Asparagir has quit IRC (Read error: Operation timed out)
03:28 🔗 Asparagir has joined #archiveteam-bs
04:23 🔗 nickname_ has quit IRC (Read error: Operation timed out)
04:25 🔗 nickname_ has joined #archiveteam-bs
04:32 🔗 Stiletto has quit IRC (Read error: Operation timed out)
04:32 🔗 Stil3tt0 has joined #archiveteam-bs
05:00 🔗 nickname_ has quit IRC (Read error: Operation timed out)
05:12 🔗 ndizzle has joined #archiveteam-bs
05:18 🔗 Somebody2 has quit IRC (Read error: Operation timed out)
05:19 🔗 ndiddy has quit IRC (Read error: Operation timed out)
05:35 🔗 Somebody2 has joined #archiveteam-bs
05:45 🔗 ndiddy has joined #archiveteam-bs
05:45 🔗 Sk1d has quit IRC (Ping timeout: 250 seconds)
05:48 🔗 ndizzle has quit IRC (Ping timeout: 244 seconds)
05:50 🔗 godane i'm uploading rev copies of UN Daily Radio
05:51 🔗 godane they do revison copies sometimes to update there radio program
05:52 🔗 Sk1d has joined #archiveteam-bs
06:17 🔗 ravetcofx has joined #archiveteam-bs
06:31 🔗 Honno has joined #archiveteam-bs
07:06 🔗 Lord_Nigh <godane> looks like ftp://aftp.cmdl.noaa.gov/ is gone <- gone? the appeal for backing it up was on reddit maybe 10 hours ago. i think everyone from /r/datahoarders tried to wget it at once
07:09 🔗 Frogging oh, that may not have been a good idea
07:15 🔗 Aranje has quit IRC (Quit: Three sheets to the wind)
07:18 🔗 Lord_Nigh do we have a complete copy of it? iirc it was started archiving in late december?
07:18 🔗 Lord_Nigh so we should have a copy of it
07:23 🔗 Stil3tt0 is now known as Stiletto
07:24 🔗 pikhq has quit IRC (Read error: Operation timed out)
07:30 🔗 pikhq has joined #archiveteam-bs
07:40 🔗 atomicthu has quit IRC (hub.dk irc.homelien.no)
07:40 🔗 PotcFdk has quit IRC (hub.dk irc.homelien.no)
07:40 🔗 alfie has quit IRC (hub.dk irc.homelien.no)
08:11 🔗 alfie has joined #archiveteam-bs
08:21 🔗 GE has joined #archiveteam-bs
08:45 🔗 SketchCow https://archive.org/details/archiveteam_ftpgov?sort=-publicdate&and[]=aftp.cmdl.noaa.gov
08:45 🔗 SketchCow We have a lot of it. arkiver will know
08:47 🔗 SketchCow I think they DDOSed it
08:48 🔗 SketchCow root@teamarchive0:/0/GODANERATOR# ftp aftp.cmdl.noaa.gov
08:48 🔗 SketchCow Connected to aftp.cmdl.noaa.gov.
08:48 🔗 SketchCow 421 There are too many connected users, please try later.
08:48 🔗 SketchCow ftp>
08:53 🔗 zhongfu has joined #archiveteam-bs
09:25 🔗 bwn has quit IRC (Read error: Operation timed out)
09:26 🔗 ravetcofx has quit IRC (Read error: Operation timed out)
09:35 🔗 Lord_Nigh SketchCow: well, based on http://www.sciencemag.org/news/2017/01/trump-officials-suspend-plan-delete-epa-climate-web-page?utm_source=newsfromscience&utm_medium=facebook-text&utm_campaign=suspendepa-10685 i don't know if it will go down anytime soon, so maybe wait for people to get bored of archiving, or ask people on reddit to nicely stop hammering the server especially with multiple connections
09:36 🔗 Lord_Nigh https://www.reddit.com/r/DataHoarder/comments/5q4xxe/erik_fichtner_on_twitter_please_wget_m_np/?utm_content=comments&utm_medium=hot&utm_source=reddit&utm_name=DataHoarder is the post about it
09:36 🔗 Lord_Nigh but its spawned by a tweet, not from reddit itself
09:36 🔗 Lord_Nigh hammering the server into oblivion just means nobody gets the data
09:36 🔗 Lord_Nigh arkiver: ^
09:37 🔗 Lord_Nigh https://www.reddit.com/r/DataHoarder/comments/5q4xxe/erik_fichtner_on_twitter_please_wget_m_np/dcwurts/ might be a good person to contact if they aren't already in here
09:38 🔗 Lord_Nigh since that might be everything, unless they're actively mirroring more
09:38 🔗 Lord_Nigh also someone on news.ycombinator.com thread about this
09:39 🔗 Lord_Nigh https://news.ycombinator.com/item?id=13487843
09:39 🔗 Lord_Nigh said they have an internet2 campus link pulling the data as well
09:48 🔗 SketchCow This is a lot of talking for a simple thing
09:48 🔗 GE has quit IRC (Remote host closed the connection)
09:54 🔗 Lord_Nigh SketchCow: ok, do we have a plan? we can continue probing the ftp periodically until a slot opens up?
10:13 🔗 antomatic has quit IRC (Read error: Operation timed out)
10:14 🔗 antomatic has joined #archiveteam-bs
10:14 🔗 swebb sets mode: +o antomatic
10:15 🔗 Coderjoe has quit IRC (Read error: Operation timed out)
10:22 🔗 Coderjoe has joined #archiveteam-bs
11:02 🔗 kniffy has quit IRC (Ping timeout: 240 seconds)
11:10 🔗 kniffy has joined #archiveteam-bs
11:32 🔗 GE has joined #archiveteam-bs
12:22 🔗 SadDM has joined #archiveteam-bs
12:22 🔗 swebb sets mode: +o SadDM
12:22 🔗 tychot has quit IRC (Ping timeout: 245 seconds)
12:24 🔗 BlueMaxim has quit IRC (Read error: Operation timed out)
12:27 🔗 pizzaiolo has joined #archiveteam-bs
12:37 🔗 tychot has joined #archiveteam-bs
12:45 🔗 pizzaiolo any particular reason archiveteam.org doesn't have HTTPS?
12:46 🔗 odemg has quit IRC (Remote host closed the connection)
12:51 🔗 bwn has joined #archiveteam-bs
12:53 🔗 Honno has quit IRC (Ping timeout: 370 seconds)
12:59 🔗 Simpbrain has quit IRC (Remote host closed the connection)
14:08 🔗 Honno has joined #archiveteam-bs
14:18 🔗 odemg has joined #archiveteam-bs
14:31 🔗 atomicthu has joined #archiveteam-bs
14:31 🔗 PotcFdk has joined #archiveteam-bs
14:53 🔗 vitzli has joined #archiveteam-bs
14:56 🔗 Honno has quit IRC (Ping timeout: 370 seconds)
16:06 🔗 xmc because nobody has made it happen yet
16:07 🔗 xmc like with most things in archiveteam that are reasonable ideas, nobody's gotten a bee up their ass to do it yet
16:37 🔗 pizzaiolo xmc: who's hosting the wiki?
16:38 🔗 Frogging s/archiveteam/any volunteer efforts/
16:44 🔗 kniffy has quit IRC (Ping timeout: 240 seconds)
16:49 🔗 kniffy has joined #archiveteam-bs
17:11 🔗 schbirid2 i used caddy for the first time today, how AWESOME
17:22 🔗 ndiddy pizzaiolo: why would you need https to access a wiki
17:22 🔗 ndiddy "oh no, attackers can spy on my wiki edits!"
17:24 🔗 Frogging lol
17:25 🔗 Frogging privacy activists would disagree but yeah I don't think it matters much
17:25 🔗 ndiddy as far as i can tell the login form is a frame to an https page so there shouldn't be any security issues
17:25 🔗 Frogging also I forget who hosts the wiki
17:26 🔗 yipdw if it's the AT wiki, there's no HTTPS at any phase, login or otherwise
17:26 🔗 ndiddy yolo
17:26 🔗 yipdw there are valid reasons for wanting HTTPS to cover all requests beyond the initial login
17:27 🔗 yipdw see e.g. technique popularized by firesheep, edit integrity
17:27 🔗 yipdw fortunately these days getting a useful cert isn't too hard what with LE and all
17:27 🔗 Frogging someone could narf your session key
17:28 🔗 Frogging or just your password *shrug*
17:28 🔗 yipdw yes, that is the technique popularized by firesheep
17:28 🔗 yipdw HTTPS isn't an unreasonable request, it just hasn't really been on the priority list
17:33 🔗 yipdw also haha I like the way Intel formatted this
17:33 🔗 yipdw https://gitlab.peach-bun.com/yipdw/random-images/uploads/298582b3b25e313b88950a27915b9e5f/intel6.png
17:33 🔗 yipdw "Buy Composer Edition! IT INCLUDES NOTHING"
17:33 🔗 Frogging lmao
17:33 🔗 yipdw yeah, you have the "All Editions Feature" subheading on the other side
17:33 🔗 Frogging Improved buying experience
17:33 🔗 MrRadar Both Firefox and Chrome are planning to mark non-secure pages with password fields as explicitly non-secure. It may be a good idea to roll out HTTPS at least for the login page before then (though if you go that far you may as well go all the way)
17:34 🔗 yipdw I dunno, I guess if the idea is to make people go for the Cluster Edition, it works
17:38 🔗 Frogging Is there some sort of filesystem container that compresses the contents automatically? My IRC logs are large but easily compressible, it'd be cool if I could put them in their own filesystem that compresses them transparently
17:38 🔗 Frogging And still be easily greppable because it's transparent
17:38 🔗 MrRadar On Windows you can enable NTFS compression on a per-folder basis
17:38 🔗 hook54321 sets mode: +o Asparagir
17:38 🔗 Frogging linux
17:38 🔗 MrRadar There's probably some FUSE file system that would work
17:40 🔗 arkiver ftp://aftp.cmdl.noaa.gov/ is up for me
17:40 🔗 Frogging must have been a reddit hug of death then
17:41 🔗 arkiver yeah
17:41 🔗 arkiver it won't work when 100 people go after the same file
17:42 🔗 arkiver that's why we have the warrior.
17:42 🔗 yipdw btrfs does transparent compression via lzo
17:42 🔗 * arkiver is afk for a bit
17:42 🔗 yipdw I dunno if you want to switch to btrfs though
17:42 🔗 arkiver when I'm back I'll check if we have all of aftp
17:42 🔗 rocode btrfs will be great when it is stable.
17:42 🔗 yipdw not that it's necessarily unstable (I have no idea), but switching filesystems is generally just a pain in the ass no matter what the target
17:43 🔗 n00b709 has joined #archiveteam-bs
17:43 🔗 Frogging yipdw: it'd just be a mounted image that sits inside my normal filesystem,
17:44 🔗 yipdw I guess that works, yeah
17:44 🔗 n00b709 has left
17:45 🔗 yipdw I never tried using different filesystems for LVM volume groups
17:45 🔗 yipdw maybe I should see if that works
17:45 🔗 yipdw you know, the next time I decide "Wow, I have nothing better to do than destroy my computer"
17:47 🔗 rocode "I will just do this one thing, how bad can it be? There is even a tutorial!" -> 5 hours later -> "Okay, I have managed to reflash by bios and unbrick my system. Let's never do that again." -> Repeat.
17:47 🔗 MrRadar https://xkcd.com/349/
17:48 🔗 yipdw yeah, basically
17:48 🔗 yipdw although, to be fair, I have had very good experiences with LVM
17:48 🔗 yipdw it was very handy when I was moving from 2 80 GB SATA SSDs to a 400 GB PCIe
17:48 🔗 yipdw add the new drive to a volume group, copy the two SSD VGs to the new one
17:48 🔗 yipdw done
17:49 🔗 yipdw I was surprised. I was expecting to have to finagle filesystem arcana or some shit and I was disappointed, in a way, that I didn't, because it meant I couldn't make snide "In 2016, ..." tweets
17:50 🔗 jrwr has joined #archiveteam-bs
17:59 🔗 kurt|rbx1 has quit IRC (Ping timeout: 260 seconds)
18:02 🔗 Honno has joined #archiveteam-bs
18:12 🔗 mhazinsk Frogging: ZFS does compression
18:19 🔗 vitzli has quit IRC (Quit: Leaving)
18:20 🔗 odemg has quit IRC (Remote host closed the connection)
18:36 🔗 kniffy has quit IRC (Ping timeout: 260 seconds)
18:42 🔗 kniffy has joined #archiveteam-bs
18:42 🔗 zino ZFS with lz4 is better than sliced break.
18:48 🔗 VADemon has joined #archiveteam-bs
18:59 🔗 odemg has joined #archiveteam-bs
19:05 🔗 kniffy has quit IRC (Ping timeout: 240 seconds)
19:06 🔗 Lord_Nigh https://www.reddit.com/r/DataHoarder/comments/5q4xxe/erik_fichtner_on_twitter_please_wget_m_np/dcxq9n9/ is worrying
19:06 🔗 Lord_Nigh judging by the warrior stats we only have at most 14gb from that ftp
19:07 🔗 Lord_Nigh while the person parent post to the one i linked on reddit has 514GB
19:09 🔗 Lord_Nigh SketchCow: maybe ask that /u/fuckoffplsthankyou guy if he can upload the 514GB to an AT machine?
19:11 🔗 kniffy has joined #archiveteam-bs
19:15 🔗 merp has joined #archiveteam-bs
19:21 🔗 merp has left
19:28 🔗 kniffy has quit IRC (Ping timeout: 240 seconds)
19:33 🔗 kniffy has joined #archiveteam-bs
19:52 🔗 Jordan has quit IRC (Read error: Operation timed out)
19:53 🔗 gourgastl has joined #archiveteam-bs
19:54 🔗 Jordan has joined #archiveteam-bs
20:03 🔗 Jordan has quit IRC (Remote host closed the connection)
20:04 🔗 Jordan has joined #archiveteam-bs
20:23 🔗 ravetcofx has joined #archiveteam-bs
20:31 🔗 kevinr has joined #archiveteam-bs
20:41 🔗 godane has quit IRC (Read error: Operation timed out)
20:48 🔗 nickname_ has joined #archiveteam-bs
21:24 🔗 alembic anybody recommend good bang/$ when it comes to VPS providers to run warrior on. I'm running on DigitalOcean atm with moderate success.
21:26 🔗 Kaz depending on the budget, but OVH's SoYouStart is pretty much the best you're going to get (not VPSs though, dedis)
21:27 🔗 alembic hmpf... might have to pay 15% sales tax on that since I'm Canadian, but 2GB for $7/mo is very competative.
21:29 🔗 gourgastl has quit IRC (Quit: Page closed)
21:31 🔗 alembic oops, that was the Kimsufi line
21:33 🔗 Kaz RAM isn't everything though, those Atom processors will be terrible for, well, pretty much everything
21:35 🔗 alembic ahaha yah, but the scripts are mostly I/O bound, no? The load wouldn't be so crazy as to bottleneck the I/O, would it?
21:36 🔗 Kaz on those processors I'd say the CPU would be your issue
21:37 🔗 Kaz depends on the project really. I'm maxing out 8 cores of a Xeon E3 on ftp-gov, but that is excessively heavy on cpu
21:39 🔗 rocode ^ I use Scaleway VPS for mine. $10 a month for 6 x x86 cores, 8GB of RAM, 200GB of SSD, 200mbit unmetered.
21:40 🔗 godane has joined #archiveteam-bs
21:43 🔗 Kaz rocode: have you tried ftp-gov on that? wondering about performance
21:45 🔗 rocode Mine is currently running 2 concurrent of the following projects: wikiteam, urlshort, yuku, pdf, googlecode, ftp, vine, ipernity, yahooanswers, and 5 grab-site grabbers.
21:45 🔗 rocode So, I am not running FTPgov, but I don't think it would be a problem. Something caused me not to run it, probably an error of some sort.
21:46 🔗 Kaz ah
21:46 🔗 Kaz just wondering as I'm having 'issues' with online/scaleway's network
21:47 🔗 rocode Tried switching to the Amsterdam datacenter?
21:47 🔗 Kaz can't, as it's a dedi through online.net rather than scaleway
21:48 🔗 rocode Ah. No clue then.
21:48 🔗 rocode I have never had network issues.
21:48 🔗 Kaz for reference, 107ms from OVH in Roubaix to one of the nasa FTPs. From online in paris I'm seeing an average of 1270ms :(
21:51 🔗 HCross2 Kaz: very simple solution. More ovh :p
21:51 🔗 Ravenloft has quit IRC (Read error: Connection reset by peer)
21:53 🔗 Kaz I'm enjoying the �20/mo I'm already saving!
21:53 🔗 Kaz shame about the 260mbps limit
21:54 🔗 HCross id use hetzner in a heartbeat if it wasnt for their bw caps
21:54 🔗 Kaz turns out we're not actually the ideal customer for any provider, who knew?
21:55 🔗 Kaz OVH could do away with Kimsufi though, bring the price down for the rest of us
21:55 🔗 HCross Kaz, tempted to test servdiscount - however from what I read their network can be slow
21:57 🔗 Kaz They have a 30day trial/moneyback thing
21:57 🔗 Kaz it's win/win. The network either works and you keep it, or it's crap so they have no reason not to honor it
21:57 🔗 HCross https://servdiscount.com/en/services/payment-methods.html is not that good tho
21:58 🔗 Kaz ?
21:58 🔗 Kaz visa/mastercard accepted
21:58 🔗 HCross for a 5% fee
21:58 🔗 HCross if you scroll down
21:58 🔗 Kaz ah, yeah they hide that a bit out the way
21:58 🔗 HCross Can do SEPA, but my bank charge £4 per payment on it
21:59 🔗 HCross although 2 eur on a 40eur server isnt too bad
22:02 🔗 Kaz http://www.speedtest.net/result/4242574225.png
22:02 🔗 Kaz admittedly that's old
22:02 🔗 HCross wonderful - the order form has suddenly gone all german on me
22:02 🔗 Kaz hah
22:03 🔗 HCross the url has /en/ in it - yet its gone all German
22:03 🔗 Kaz https://www.peeringdb.com/net/1007 too - doesn't seem too bad but doubt you'll get anything worthwhile outside of europe
22:03 🔗 HCross im pretty amazed with my box in LA. 140ms between my home and the server, RDP still feels like its at OVH in france, and gameservers and stuff run well
22:03 🔗 BartoCH can someone requeue the flickr items or change the warrior default to something else? I'm feeling like it doens't do much right now.
22:06 🔗 HCross Kaz, their order form cant take a full UK postcode
22:10 🔗 Kaz ah
22:14 🔗 alembic i've noticed ftp-gov has been heavy on CPU as well Kaz, any idea why? Also, why does the item have to fit in memory?
22:14 🔗 alembic if it's a matter of having to send it off to the AT servers, then we should be able to read and send in chunks, no?
22:16 🔗 Kaz honestly no idea, I haven't looked much into what the scripts are actually doing other than pointing wpull at some ftp sites
22:17 🔗 yipdw the ftp-gov scripts use wpull's on-disk URL database; if there's a memory limit somewhere, at least it isn't that
22:20 🔗 alembic k
22:21 🔗 yipdw as for high CPU, we notice the same phenomenon in archivebot, which is also using wpull
22:21 🔗 yipdw the cause is presently unknown because we haven't profiled yet
22:22 🔗 yipdw in archivebot's case it's possible it's not wpull proper but rather one of our plugins
22:22 🔗 yipdw etc.
22:23 🔗 HCross yipdw, are you taking pipeline applications?
22:23 🔗 yipdw maybe soon. I want to keep watching the new pipeline code for a bit
22:24 🔗 alembic coolio... I've been getting a lot of memory surprises with long-running python scripts these days myself
22:24 🔗 alembic one time it was a memory leak from a c library interface, the other time it was one of those weird subotimal GC cases
22:25 🔗 yipdw we usually don't hit memory problems in pipelines
22:25 🔗 yipdw although sometimes it happens
22:25 🔗 yipdw it's odd
22:25 🔗 yipdw i'd really love a visualvm equivalent for CPython
22:26 🔗 alembic there are some weird profiling tools for python, but it's all opaque to me
22:28 🔗 yipdw there are a few but the ones I've seen give you a flat or graph profile at the end of some block, or at process termination. i want the ability (either by explicit instrumentation or VM hooks) to watch the process as it runs
22:28 🔗 yipdw there's so much information you can get about process behavior by watching it over time vs. trying to reconstruct that history from a profiler snapshot. it's strange that this isn't more common in profiler land
22:29 🔗 yipdw like offhand there's visualvm, instruments, telemetry, uh
22:29 🔗 yipdw vtune maybe
22:42 🔗 dashcloud yipdw: you may have seen this already, but just in case you haven't, is this helpful: http://www.brendangregg.com/blog/2016-10-27/dtrace-for-linux-2016.html (the person who worked a lot on bpf)
22:44 🔗 yipdw dtrace is pretty nice and there is some work on integrating dtrace probes into CPython, yeah
22:44 🔗 yipdw I hope that takes off
22:44 🔗 yipdw I didn't know DTrace made it into Linux
22:45 🔗 yipdw oh, wait, it didn't
22:45 🔗 yipdw well maybe one day
22:56 🔗 odemg has quit IRC (Remote host closed the connection)
23:02 🔗 GE has quit IRC (Quit: zzz)
23:05 🔗 Honno has quit IRC (Ping timeout: 370 seconds)
23:20 🔗 odemg has joined #archiveteam-bs
23:23 🔗 dashcloud from the post, it sounds like bpf offers everything dtrace does except the vast library of pre-existing probes
23:25 🔗 Stiletto has quit IRC (Read error: Operation timed out)
23:26 🔗 dashcloud has quit IRC (Read error: Operation timed out)
23:32 🔗 dashcloud has joined #archiveteam-bs
23:37 🔗 BlueMaxim has joined #archiveteam-bs
23:55 🔗 Stil3tt0 has joined #archiveteam-bs

irclogger-viewer