#warrior 2019-03-26,Tue

↑back Search

Time Nickname Message
02:41 🔗 warmwaffl has quit IRC (Remote host closed the connection)
03:22 🔗 bithippo has joined #warrior
03:43 🔗 bithippo has quit IRC (Textual IRC Client: www.textualapp.com)
05:52 🔗 Atom__ has quit IRC (Read error: Connection reset by peer)
05:56 🔗 Atom__ has joined #warrior
13:10 🔗 minkafigh has joined #warrior
13:16 🔗 minkafigh hello, anybody here? I have a vServer on Digitalocean and am currently archiving google+. I only download with 30-60MBit/s though, why is that? My vServer can easily do 500mbit/s down and 700mbit/s up.
13:16 🔗 MrRadar2 How many concurrent are you running?
13:17 🔗 minkafigh 6
13:17 🔗 MrRadar2 Each thread downloads 1 page at a time and much of that time is taken up by connecting to Google's servers
13:17 🔗 MrRadar2 In other words it's normal.
13:18 🔗 minkafigh is there a way to speed up that connecting?
13:18 🔗 minkafigh are there like, datacenters that are faster than others?
13:18 🔗 minkafigh in terms of connecting to google
13:18 🔗 MrRadar2 Yes, but I'm not sure which ones are best
13:18 🔗 MrRadar2 Additionally, with Google+ the main bottleneck right now is actually our upload capacity, not download
13:19 🔗 minkafigh Why is that?
13:19 🔗 MrRadar2 If you want to increase your throughput the best way to do it is to run with a higher concurrent
13:19 🔗 MrRadar2 Our upload staging servers are completely full and the Internet Archive's upload service is pretty much fully maxed out
13:19 🔗 minkafigh If you run higher concurrent i face a error that says that i have too many connections reached
13:20 🔗 MrRadar2 Is that an rsync error? If so, that's normal
13:20 🔗 MrRadar2 That's due to the aformentioned upload bottleneck
13:22 🔗 minkafigh How much storage will google+ need?
13:23 🔗 MrRadar2 We're currently at 800 TB
13:23 🔗 MrRadar2 It's looking like it will be over 1 petabyte total
13:23 🔗 MrRadar2 You can check the stats here: http://tracker.archiveteam.org/googleplus/
13:24 🔗 minkafigh How can Fusl upload that fast when the upload servers are full?
13:24 🔗 minkafigh does he just have very many servers?
13:25 🔗 MrRadar2 Yes, exactly. She's running tons of servers
13:25 🔗 MrRadar2 Including several of the upload servers
13:25 🔗 Atom-- has joined #warrior
13:26 🔗 minkafigh Do I just send my stuff to the upload servers and then those servers are uploading it to archive.org or what are they doing?
13:26 🔗 Kaz correct
13:26 🔗 MrRadar2 Yes, that's exactly what's happening
13:26 🔗 minkafigh Why have those "middle mans"?
13:27 🔗 MrRadar2 The servers you are uploading to combine all of the smaller files you send into 50 GB "megawarcs" that are easier for the IA's infrastructure to handle
13:27 🔗 minkafigh damn :O
13:28 🔗 minkafigh How much time do we have till google+ is closed?
13:28 🔗 minkafigh And can we reach the goal to archive everything?
13:28 🔗 minkafigh with out current speed
13:28 🔗 JAA The channel for Google+ is #googleminus .
13:28 🔗 Atom__ has quit IRC (Ping timeout: 252 seconds)
15:24 🔗 minkafigh has quit IRC (Quit: Page closed)
15:24 🔗 minkafigh has joined #warrior
15:24 🔗 minkafigh has quit IRC (Client Quit)
15:29 🔗 minkafigh has joined #warrior
17:00 🔗 tmg1|eva interestingly after the last seasaw update i haven't had the virtualbox stop processing on me in its entirety...but 2 of the workers(?) within one virtualbox instance have stalled
17:03 🔗 MrRadar2 What is the last output from those workers?
17:05 🔗 tmg1|eva 1885=200 https://lh3.googleusercontent.com/-kMl27Cqwj6U/VxkQ-QgQL5I/AAAAAAAAOig/1-yZcxtUOwMbAc7hTSpObLvBR8RzN-XhA/w530-h636-n/2016%2B-%2B1
17:05 🔗 tmg1|eva and 141=200 https://plus.google.com/113183347162063862096/posts/aPn7KwAgSf8
17:05 🔗 tmg1|eva both of which are real posts that exist but are not that large
17:25 🔗 Kaz not a seesaw issue I don't think
17:25 🔗 JAA I know I had wget-lua get stuck before. This was two years ago though.
17:31 🔗 Kaz Yeah, it definitely happens, but I'd expect to see more of it than just this one event. Would blame local issues first (seriously, maybe it's just slow) rather than assume seesaw/wget-lua issues
17:31 🔗 JAA Yeah
17:34 🔗 robbierut 1 of my vm's totally froze, couldnt enter the webinterface (was running headless vm) and for some reason ate all my disk io. Now I stopped it.
17:35 🔗 robbierut Another vm keeps restarting every time, after I shut it down via the soft close web button
17:53 🔗 tmg1|eva Kaz: it's not stuck for 2 days slow.
17:53 🔗 Kaz ctrl-c and see what happens
17:54 🔗 Kaz or in fact if it's warrior you can't even do that
17:54 🔗 tmg1|eva Kaz: also, up until these 2 workers froze my vm freezes about once every day or two in its entirety up until yesterday - this is the first time only 2 (and not all 6) workers stopped processing
17:55 🔗 tmg1|eva i'll try ctrl-c if the rest stop
17:56 🔗 tmg1|eva i mean could ctrl-c on the vm main screen
20:10 🔗 minkafigh has quit IRC (Quit: Page closed)
20:37 🔗 warmwaffl has joined #warrior
21:49 🔗 _54674341 has joined #warrior
22:26 🔗 netsound has quit IRC (Read error: Connection reset by peer)
22:27 🔗 netsound has joined #warrior
23:25 🔗 VADemon has joined #warrior

irclogger-viewer