| Time |
Nickname |
Message |
|
00:11
🔗
|
|
ephemer0l has quit IRC (Ping timeout: 745 seconds) |
|
00:17
🔗
|
|
killsushi has joined #archiveteam-ot |
|
00:20
🔗
|
|
ephemer0l has joined #archiveteam-ot |
|
02:20
🔗
|
|
ShellyRol has quit IRC (Read error: Connection reset by peer) |
|
02:20
🔗
|
|
ShellyRol has joined #archiveteam-ot |
|
02:34
🔗
|
Somebody2 |
Oh, phuzion dropped by! That's a name I haven't seen in a while... :-) |
|
02:35
🔗
|
phuzion |
Somebody2: I'm idling in here all the time lol |
|
02:38
🔗
|
Somebody2 |
yeah, so am I, but neither of us actually *say* anything very often... |
|
02:39
🔗
|
|
bluefoo_ has quit IRC (Quit: bluefoo_) |
|
02:45
🔗
|
|
lunik1498 has quit IRC (Read error: Operation timed out) |
|
03:46
🔗
|
|
qw3rty114 has joined #archiveteam-ot |
|
03:52
🔗
|
|
qw3rty113 has quit IRC (Read error: Operation timed out) |
|
04:05
🔗
|
|
godane has quit IRC (Read error: Operation timed out) |
|
04:10
🔗
|
|
killsushi has quit IRC (Quit: Leaving) |
|
04:11
🔗
|
|
killsushi has joined #archiveteam-ot |
|
04:34
🔗
|
|
Flashfloo has quit IRC (Remote host closed the connection) |
|
04:34
🔗
|
|
Flashfire has quit IRC (Remote host closed the connection) |
|
04:34
🔗
|
|
kiska has quit IRC (Remote host closed the connection) |
|
04:35
🔗
|
|
Flashfloo has joined #archiveteam-ot |
|
04:35
🔗
|
|
Flashfire has joined #archiveteam-ot |
|
04:35
🔗
|
|
kiska has joined #archiveteam-ot |
|
04:35
🔗
|
|
Fusl__ sets mode: +o kiska |
|
04:35
🔗
|
|
Fusl sets mode: +o kiska |
|
04:35
🔗
|
|
Fusl_ sets mode: +o kiska |
|
05:11
🔗
|
|
fuzzy8021 has quit IRC (Read error: Connection reset by peer) |
|
05:11
🔗
|
|
fuzzy8021 has joined #archiveteam-ot |
|
05:14
🔗
|
kpcyrd |
does somebody happen to know if there's an rss feed for archive.org users? |
|
06:25
🔗
|
|
yano has quit IRC (west.us.hub irc.mzima.net) |
|
06:25
🔗
|
|
eientei95 has quit IRC (west.us.hub irc.mzima.net) |
|
06:25
🔗
|
|
wp494 has quit IRC (west.us.hub irc.mzima.net) |
|
06:25
🔗
|
|
eythian has quit IRC (west.us.hub irc.mzima.net) |
|
06:25
🔗
|
|
svchfoo3 has quit IRC (west.us.hub irc.mzima.net) |
|
06:25
🔗
|
|
nyany has quit IRC (west.us.hub irc.mzima.net) |
|
06:26
🔗
|
|
yano has joined #archiveteam-ot |
|
06:26
🔗
|
|
eientei95 has joined #archiveteam-ot |
|
06:26
🔗
|
|
wp494 has joined #archiveteam-ot |
|
06:26
🔗
|
|
eythian has joined #archiveteam-ot |
|
06:26
🔗
|
|
svchfoo3 has joined #archiveteam-ot |
|
06:26
🔗
|
|
nyany has joined #archiveteam-ot |
|
06:26
🔗
|
|
irc.mzima.net sets mode: +o svchfoo3 |
|
06:26
🔗
|
|
Fusl__ sets mode: +o svchfoo3 |
|
06:26
🔗
|
|
Fusl sets mode: +o svchfoo3 |
|
06:26
🔗
|
|
Fusl_ sets mode: +o svchfoo3 |
|
09:59
🔗
|
|
m007a83 has joined #archiveteam-ot |
|
10:16
🔗
|
|
BlueMax has quit IRC (Read error: Connection reset by peer) |
|
11:46
🔗
|
|
Ravenloft has quit IRC (Read error: Connection reset by peer) |
|
12:06
🔗
|
eythian |
I have a VPS for archivy things running at scaleway (where I already have a server, so it was easy), however it is the cheapest CPU level and as far as I can tell, I can only have a max 50GB disk on it. I've found that it's pretty easy for this to fill up in some things. Does anyone have any recommendations for either a similarly priced but more flexible provider, or a way to attach more disk to smaller specced servers at scaleway? |
|
12:07
🔗
|
Igloo |
Digital Ocean? |
|
12:07
🔗
|
Igloo |
I don't know scaleway's pricing |
|
12:10
🔗
|
eythian |
I can't even find it on their pricing list any more, but it's 2CPU, 2GB RAM, 50GB disk for €3,99/mo |
|
12:12
🔗
|
ivan_ |
are you running grab-site on it? |
|
12:14
🔗
|
Igloo |
https://www.hetzner.com/cloud are good too |
|
12:14
🔗
|
eythian |
yeah, I was also running the newsgrabber. |
|
12:14
🔗
|
Igloo |
Newsgrabber is huge. |
|
12:14
🔗
|
eythian |
so I discovered :) |
|
12:15
🔗
|
yano |
i thought newsgrabber wasn't currently active? |
|
12:15
🔗
|
eythian |
actually, the console interface for scaleway suggests I can add more disk, which I thought it didn't allow last time I looked. |
|
12:16
🔗
|
eythian |
I might have to try that. |
|
12:17
🔗
|
eythian |
yano: I think it's semi active? you can run it manually or something. It was a little while ago that I tried. |
|
12:17
🔗
|
Igloo |
yano it's not. |
|
12:17
🔗
|
Igloo |
We killed off whilst we fix some issues |
|
12:25
🔗
|
yano |
ah, gotcha |
|
12:25
🔗
|
yano |
does newsgrabber focus on specific new sources? or any news articles? |
|
12:26
🔗
|
yano |
i run ##news on freenode and i have the history of that channel going back years |
|
12:26
🔗
|
yano |
it pulls in over 600+ rss feeds from world-wide news sources |
|
12:30
🔗
|
eythian |
oh, that's neat |
|
12:31
🔗
|
Igloo |
It uses ~1,000 sources |
|
12:31
🔗
|
Igloo |
Not just RSS |
|
12:31
🔗
|
Igloo |
https://github.com/ArchiveTeam/NewsGrabber-Services |
|
12:54
🔗
|
yano |
Igloo: cool |
|
12:56
🔗
|
yano |
hm, i went to manually sign up for NewsGrabber and I get "Project code is out of date and needs to be upgraded. To remedy this problem immediately, you may reboot your warrior. Retrying after 170 seconds..." I rebooted the docker container and still the same thing, do I need to rebuild it? or is the tracker kind of on pause/hold? |
|
12:56
🔗
|
Igloo |
Yeah, It's paused |
|
12:56
🔗
|
Igloo |
So it won't issue anything until we fix some issues |
|
12:57
🔗
|
yano |
gotcha |
|
13:18
🔗
|
|
bluefoo has joined #archiveteam-ot |
|
13:27
🔗
|
JAA |
eythian: You can add more disk to a Scaleway server, but it's a separate volume that you then somehow need to merge together. And I believe you'll run into issues if you try to combine the primary and secondary volumes, so you might need to keep the primary one for just the base system and buy two or more secondary ones for the scripts dir. |
|
13:28
🔗
|
eythian |
JAA: yeah, I have no problem LVMing some volumes together, it was just that when I last looked you couldn't add volumes to the lower-end servers. This appears to not be the case now though. |
|
13:29
🔗
|
JAA |
Ah, right. |
|
13:43
🔗
|
|
killsushi has quit IRC (Quit: Leaving) |
|
13:44
🔗
|
|
killsushi has joined #archiveteam-ot |
|
13:48
🔗
|
|
dhyan_nat has joined #archiveteam-ot |
|
14:10
🔗
|
|
dhyan_nat has quit IRC (Ping timeout: 864 seconds) |
|
14:20
🔗
|
|
kiska1 has quit IRC (Remote host closed the connection) |
|
14:20
🔗
|
|
kiska1 has joined #archiveteam-ot |
|
14:20
🔗
|
|
Fusl__ sets mode: +o kiska1 |
|
14:20
🔗
|
|
Fusl sets mode: +o kiska1 |
|
14:20
🔗
|
|
Fusl_ sets mode: +o kiska1 |
|
15:24
🔗
|
kiska |
I had great fun working with scaleway instances during #googleminus (Read: NOT AT ALL) |
|
15:25
🔗
|
Fusl |
/s |
|
15:30
🔗
|
kiska |
A few incidences... https://media.discordapp.net/attachments/553892314590871562/561771438546223106/unknown.png |
|
15:31
🔗
|
Fusl |
mhm |
|
15:31
🔗
|
Fusl |
see |
|
15:31
🔗
|
Fusl |
thats why the docker pipeline exists |
|
15:31
🔗
|
Fusl |
to avoid all of this |
|
15:31
🔗
|
Fusl |
and yet they go the non-docker way |
|
15:31
🔗
|
Fusl |
and fuck it up every single time there is a new project |
|
15:32
🔗
|
kiska |
That was before the docker pipeline code existed |
|
15:32
🔗
|
Fusl |
there is at least one person who misconfigures their megawarc factory. every. single. god. damn. time. |
|
15:32
🔗
|
Fusl |
doesn't matter, you all still make the same mistake :P |
|
15:32
🔗
|
kiska |
But the issue here was scaleway unmounting my volume |
|
15:32
🔗
|
Fusl |
you learned nothing from those incidents |
|
15:32
🔗
|
Fusl |
yes |
|
15:32
🔗
|
Fusl |
and you /STILL/ use scaleway |
|
15:32
🔗
|
kiska |
xD |
|
15:32
🔗
|
Fusl |
even though you know its shit |
|
15:32
🔗
|
kpcyrd |
I had to switch creditcards once due to changes at my bank. didn't have any issues re-adding it everywhere, except scaleway, even after writing support the reply was basically that they would rather not take my money anymore |
|
15:33
🔗
|
kpcyrd |
I think there was also a minor hickup at vultr, but their support quickly fixed the issue and apologized |
|
15:33
🔗
|
Fusl |
vultr is very open with network and server problems they have. they even send out an email if your instance gets rebooted due to a hardware malfunction |
|
15:34
🔗
|
Fusl |
same as linode |
|
15:34
🔗
|
kiska |
Hey at that time we were trying to get our hands on as many gbit servers for as little price as we could |
|
15:34
🔗
|
Fusl |
hetzner cloud my friend |
|
15:35
🔗
|
kpcyrd |
I also moved almost everything to hetzner cloud over time |
|
15:35
🔗
|
Fusl |
lol i'm moving all my stuff away from hetzner cloud do on-premises hardware |
|
15:35
🔗
|
kiska |
But hey when scaleway works, I can slam 2.5gbps to a server |
|
15:35
🔗
|
Fusl |
when hetzner works you can slam 10gbps to a server |
|
15:35
🔗
|
Fusl |
;) |
|
15:35
🔗
|
kiska |
With a 30T cap |
|
15:35
🔗
|
Fusl |
so? |
|
15:35
🔗
|
kpcyrd |
per instance |
|
15:36
🔗
|
Fusl |
per instance launch |
|
15:36
🔗
|
kiska |
Yeah not paying 1 Eur/T of bandwidth |
|
15:36
🔗
|
Fusl |
terminate the instance and get a new one, problem solved |
|
15:36
🔗
|
Fusl |
btw hetzner |
|
15:36
🔗
|
Fusl |
there is the hetzner forum |
|
15:37
🔗
|
Fusl |
which requires a login to work |
|
15:37
🔗
|
Fusl |
have we ever considered doing a grab-site of it? |
|
15:37
🔗
|
kiska |
https://media.discordapp.net/attachments/553892314590871562/557783192694685696/unknown.png Scaleway C2L instance xD |
|
15:38
🔗
|
kpcyrd |
the instance launch trick is nice, I'm currently using dedicated hetzner boxes for high traffic usecases since they are all unmetered |
|
15:38
🔗
|
Fusl |
you can also just proxy your rsync connections through your dedicated servers |
|
15:38
🔗
|
kiska |
I want to be able to do the instance launch trick with their dedicated stuff, but nope you can't since they have a 30 day warning |
|
15:38
🔗
|
Fusl |
to get unmetered traffic |
|
15:38
🔗
|
Fusl |
kiska: dedicated is fully unmetered |
|
15:39
🔗
|
kiska |
10gbit |
|
15:39
🔗
|
Fusl |
oh |
|
15:39
🔗
|
Fusl |
:D |
|
15:39
🔗
|
kiska |
Should have made that clear |
|
15:39
🔗
|
Fusl |
well |
|
15:39
🔗
|
Fusl |
you can always get a server and then cancel it within 14 days |
|
15:39
🔗
|
Fusl |
but i dont think thats true for a 10g upgrade then |
|
15:39
🔗
|
kiska |
Yeah.... |
|
15:39
🔗
|
|
bluefoo has quit IRC (Read error: Connection reset by peer) |
|
15:40
🔗
|
Fusl |
in any case, it wouldnt help you anyways. i dont even get a full 1gbps to IA from a single server, so how would 10gbps help? |
|
15:40
🔗
|
kiska |
The ingest, egress we can worry about later... until disks fill |
|
15:40
🔗
|
Fusl |
get a sx292 server |
|
15:40
🔗
|
Fusl |
with 10gbps |
|
15:41
🔗
|
Fusl |
accept connections through cloud servers, buffer the data |
|
15:41
🔗
|
Fusl |
then quickly move the files over to the 10g server |
|
15:41
🔗
|
Fusl |
have another set of cloud servers read that data from the 10g servers and immediately delete them there |
|
15:41
🔗
|
Fusl |
run the factory on those cloud servers |
|
15:41
🔗
|
Fusl |
and then directly upload to IA |
|
15:42
🔗
|
kiska |
https://media.discordapp.net/attachments/553892314590871562/557528410629603338/unknown.png Since this no longer exists, I am fine with posting it here |
|
15:43
🔗
|
Fusl |
that looks similarly bad to abox fks1 :D https://atdash.meo.ws/d/000000058/telegraf-detail?var-user=fusl&var-host=archivebox-fks1 |
|
15:43
🔗
|
|
bluefoo has joined #archiveteam-ot |
|
15:44
🔗
|
kiska |
Where did all my bandwidth go! |
|
15:48
🔗
|
kpcyrd |
btw, are the urlteam uploads to archive.org incremental or the whole thing every time? |
|
15:53
🔗
|
Fusl |
incremental |
|
15:58
🔗
|
Dallas |
@kiska what command is that? |
|
15:59
🔗
|
kiska |
nload -i 1000000 -o 1000 or whatever I had it |
|
15:59
🔗
|
Dallas |
Fusl I so wanna get round to setting up dashboards like that but need tiimmee |
|
15:59
🔗
|
Dallas |
oh sick, ty |
|
16:00
🔗
|
Fusl |
Dallas: you can export the dashboard json from grafana and use it for yourself |
|
16:00
🔗
|
Fusl |
they are MIT licensed so have fun |
|
16:01
🔗
|
Dallas |
really !?! I did not know that, I may do thanks :D |
|
16:03
🔗
|
yano |
https://armis.com/urgent11/ |
|
16:05
🔗
|
kiska |
RIP |
|
16:13
🔗
|
Fusl |
"since version 6.5" |
|
16:14
🔗
|
Fusl |
yay i'm not affected? http://xor.meo.ws/77efcb2d/f7dd/4d24/8443/0499e8ce8625.png |
|
16:14
🔗
|
Fusl |
:D |
|
16:15
🔗
|
kpcyrd |
any recommendations for metrics collection for grafana (like cpu/disk/bandwidth usage)? |
|
16:16
🔗
|
Fusl |
telegraf |
|
16:27
🔗
|
Igloo |
I'm tempted to colo some stuff in America next time I go out |
|
16:27
🔗
|
Igloo |
Management of replacement disks is the only PITA (or just remote hands) |
|
17:07
🔗
|
|
Dj-Wawa has joined #archiveteam-ot |
|
17:17
🔗
|
|
Dragnog has joined #archiveteam-ot |
|
17:49
🔗
|
schbirid |
just order a handjob from one of them cam girls |
|
18:50
🔗
|
|
ShellyRol has quit IRC (Read error: Operation timed out) |
|
18:52
🔗
|
|
ShellyRol has joined #archiveteam-ot |
|
19:45
🔗
|
Fusl |
"remote hands" huh |
|
19:59
🔗
|
Igloo |
Remote monkeys. |
|
19:59
🔗
|
Igloo |
Maybe more apt. |
|
20:00
🔗
|
Igloo |
The webcite stuff |
|
20:00
🔗
|
Igloo |
Do we want it as a web collection? |
|
20:01
🔗
|
Igloo |
For WBM usage? |
|
20:13
🔗
|
Fusl |
the s3 urls? sure, why not |
|
20:16
🔗
|
Igloo |
On it's way up |
|
21:29
🔗
|
|
killsushi has quit IRC (Read error: Connection reset by peer) |
|
21:29
🔗
|
|
killsushi has joined #archiveteam-ot |
|
21:31
🔗
|
|
killsushi has quit IRC (Read error: Connection reset by peer) |
|
21:31
🔗
|
|
killsushi has joined #archiveteam-ot |
|
22:06
🔗
|
|
ShellyRol has quit IRC (Read error: Operation timed out) |
|
22:07
🔗
|
|
ShellyRol has joined #archiveteam-ot |
|
22:17
🔗
|
|
Jens has quit IRC (Remote host closed the connection) |
|
22:18
🔗
|
|
Jens has joined #archiveteam-ot |
|
22:26
🔗
|
|
JAA has quit IRC (Read error: Operation timed out) |
|
22:27
🔗
|
|
kiskabak has quit IRC (Ping timeout (120 seconds)) |
|
22:29
🔗
|
|
kiskabak has joined #archiveteam-ot |
|
22:29
🔗
|
|
Fusl__ sets mode: +o kiskabak |
|
22:29
🔗
|
|
Fusl sets mode: +o kiskabak |
|
22:29
🔗
|
|
Fusl_ sets mode: +o kiskabak |
|
22:30
🔗
|
|
JAA has joined #archiveteam-ot |
|
22:30
🔗
|
|
Fusl__ sets mode: +o JAA |
|
22:30
🔗
|
|
Fusl sets mode: +o JAA |
|
22:30
🔗
|
|
Fusl_ sets mode: +o JAA |
|
22:31
🔗
|
|
AlsoJAA sets mode: +o JAA |
|
23:13
🔗
|
ivan_ |
kpcyrd: prometheus-node-exporter? |