| Time |
Nickname |
Message |
|
02:41
🔗
|
|
warmwaffl has quit IRC (Remote host closed the connection) |
|
03:22
🔗
|
|
bithippo has joined #warrior |
|
03:43
🔗
|
|
bithippo has quit IRC (Textual IRC Client: www.textualapp.com) |
|
05:52
🔗
|
|
Atom__ has quit IRC (Read error: Connection reset by peer) |
|
05:56
🔗
|
|
Atom__ has joined #warrior |
|
13:10
🔗
|
|
minkafigh has joined #warrior |
|
13:16
🔗
|
minkafigh |
hello, anybody here? I have a vServer on Digitalocean and am currently archiving google+. I only download with 30-60MBit/s though, why is that? My vServer can easily do 500mbit/s down and 700mbit/s up. |
|
13:16
🔗
|
MrRadar2 |
How many concurrent are you running? |
|
13:17
🔗
|
minkafigh |
6 |
|
13:17
🔗
|
MrRadar2 |
Each thread downloads 1 page at a time and much of that time is taken up by connecting to Google's servers |
|
13:17
🔗
|
MrRadar2 |
In other words it's normal. |
|
13:18
🔗
|
minkafigh |
is there a way to speed up that connecting? |
|
13:18
🔗
|
minkafigh |
are there like, datacenters that are faster than others? |
|
13:18
🔗
|
minkafigh |
in terms of connecting to google |
|
13:18
🔗
|
MrRadar2 |
Yes, but I'm not sure which ones are best |
|
13:18
🔗
|
MrRadar2 |
Additionally, with Google+ the main bottleneck right now is actually our upload capacity, not download |
|
13:19
🔗
|
minkafigh |
Why is that? |
|
13:19
🔗
|
MrRadar2 |
If you want to increase your throughput the best way to do it is to run with a higher concurrent |
|
13:19
🔗
|
MrRadar2 |
Our upload staging servers are completely full and the Internet Archive's upload service is pretty much fully maxed out |
|
13:19
🔗
|
minkafigh |
If you run higher concurrent i face a error that says that i have too many connections reached |
|
13:20
🔗
|
MrRadar2 |
Is that an rsync error? If so, that's normal |
|
13:20
🔗
|
MrRadar2 |
That's due to the aformentioned upload bottleneck |
|
13:22
🔗
|
minkafigh |
How much storage will google+ need? |
|
13:23
🔗
|
MrRadar2 |
We're currently at 800 TB |
|
13:23
🔗
|
MrRadar2 |
It's looking like it will be over 1 petabyte total |
|
13:23
🔗
|
MrRadar2 |
You can check the stats here: http://tracker.archiveteam.org/googleplus/ |
|
13:24
🔗
|
minkafigh |
How can Fusl upload that fast when the upload servers are full? |
|
13:24
🔗
|
minkafigh |
does he just have very many servers? |
|
13:25
🔗
|
MrRadar2 |
Yes, exactly. She's running tons of servers |
|
13:25
🔗
|
MrRadar2 |
Including several of the upload servers |
|
13:25
🔗
|
|
Atom-- has joined #warrior |
|
13:26
🔗
|
minkafigh |
Do I just send my stuff to the upload servers and then those servers are uploading it to archive.org or what are they doing? |
|
13:26
🔗
|
Kaz |
correct |
|
13:26
🔗
|
MrRadar2 |
Yes, that's exactly what's happening |
|
13:26
🔗
|
minkafigh |
Why have those "middle mans"? |
|
13:27
🔗
|
MrRadar2 |
The servers you are uploading to combine all of the smaller files you send into 50 GB "megawarcs" that are easier for the IA's infrastructure to handle |
|
13:27
🔗
|
minkafigh |
damn :O |
|
13:28
🔗
|
minkafigh |
How much time do we have till google+ is closed? |
|
13:28
🔗
|
minkafigh |
And can we reach the goal to archive everything? |
|
13:28
🔗
|
minkafigh |
with out current speed |
|
13:28
🔗
|
JAA |
The channel for Google+ is #googleminus . |
|
13:28
🔗
|
|
Atom__ has quit IRC (Ping timeout: 252 seconds) |
|
15:24
🔗
|
|
minkafigh has quit IRC (Quit: Page closed) |
|
15:24
🔗
|
|
minkafigh has joined #warrior |
|
15:24
🔗
|
|
minkafigh has quit IRC (Client Quit) |
|
15:29
🔗
|
|
minkafigh has joined #warrior |
|
17:00
🔗
|
tmg1|eva |
interestingly after the last seasaw update i haven't had the virtualbox stop processing on me in its entirety...but 2 of the workers(?) within one virtualbox instance have stalled |
|
17:03
🔗
|
MrRadar2 |
What is the last output from those workers? |
|
17:05
🔗
|
tmg1|eva |
1885=200 https://lh3.googleusercontent.com/-kMl27Cqwj6U/VxkQ-QgQL5I/AAAAAAAAOig/1-yZcxtUOwMbAc7hTSpObLvBR8RzN-XhA/w530-h636-n/2016%2B-%2B1 |
|
17:05
🔗
|
tmg1|eva |
and 141=200 https://plus.google.com/113183347162063862096/posts/aPn7KwAgSf8 |
|
17:05
🔗
|
tmg1|eva |
both of which are real posts that exist but are not that large |
|
17:25
🔗
|
Kaz |
not a seesaw issue I don't think |
|
17:25
🔗
|
JAA |
I know I had wget-lua get stuck before. This was two years ago though. |
|
17:31
🔗
|
Kaz |
Yeah, it definitely happens, but I'd expect to see more of it than just this one event. Would blame local issues first (seriously, maybe it's just slow) rather than assume seesaw/wget-lua issues |
|
17:31
🔗
|
JAA |
Yeah |
|
17:34
🔗
|
robbierut |
1 of my vm's totally froze, couldnt enter the webinterface (was running headless vm) and for some reason ate all my disk io. Now I stopped it. |
|
17:35
🔗
|
robbierut |
Another vm keeps restarting every time, after I shut it down via the soft close web button |
|
17:53
🔗
|
tmg1|eva |
Kaz: it's not stuck for 2 days slow. |
|
17:53
🔗
|
Kaz |
ctrl-c and see what happens |
|
17:54
🔗
|
Kaz |
or in fact if it's warrior you can't even do that |
|
17:54
🔗
|
tmg1|eva |
Kaz: also, up until these 2 workers froze my vm freezes about once every day or two in its entirety up until yesterday - this is the first time only 2 (and not all 6) workers stopped processing |
|
17:55
🔗
|
tmg1|eva |
i'll try ctrl-c if the rest stop |
|
17:56
🔗
|
tmg1|eva |
i mean could ctrl-c on the vm main screen |
|
20:10
🔗
|
|
minkafigh has quit IRC (Quit: Page closed) |
|
20:37
🔗
|
|
warmwaffl has joined #warrior |
|
21:49
🔗
|
|
_54674341 has joined #warrior |
|
22:26
🔗
|
|
netsound has quit IRC (Read error: Connection reset by peer) |
|
22:27
🔗
|
|
netsound has joined #warrior |
|
23:25
🔗
|
|
VADemon has joined #warrior |