Time |
Nickname |
Message |
00:01
π
|
arkiver |
working on it now |
00:02
π
|
arkiver |
anyone wants to make a wiki page for it? |
00:09
π
|
arkiver |
JAA: shall we just get the review URLs as well :P |
00:09
π
|
arkiver |
the outlinks |
00:10
π
|
Ryz |
That can work, in a separate job yeah~ |
00:10
π
|
Ryz |
Ran the 2 jobs with no-offsite-links because of a lot of friction potential |
00:11
π
|
arkiver |
will keep it in the same warrior project |
00:12
π
|
arkiver |
only getting the HTML, no page requisites |
00:13
π
|
JAA |
Yeah, grabbing the reviews as well would be nice. Not sure if it should be done in a second phase though so we're sure we can grab the site itself in time before the shutdown. |
00:13
π
|
arkiver |
if we have a large amount of concurrent it should not be a problem |
00:13
π
|
JAA |
I know it's only a few million URLs (including the image pages etc.), but still. |
00:13
π
|
arkiver |
in case of problem on the outlinks I'll just skip them |
00:13
π
|
arkiver |
problems* |
00:15
π
|
arkiver |
do we know of any games with a very large number of reviews? |
00:15
π
|
arkiver |
possible pagination |
00:16
π
|
arkiver |
actually yeah, let's do it in a seperate grab project, we can also get the news URLs then |
00:19
π
|
JAA |
Gears of War has 108 reviews and no pagination: https://www.gamerankings.com/xbox360/928234-gears-of-war/articles.html |
00:20
π
|
JAA |
But if we just follow any link of https://www.gamerankings.com/PLATFORM/ID-SLUG/ (after detection of PLATFORM and SLUG, for IDs in the item range), any such cases should be covered. |
00:21
π
|
arkiver |
yeah, but indeed let's do that later |
00:21
π
|
arkiver |
and also get all articles |
00:21
π
|
arkiver |
the news |
00:21
π
|
JAA |
Why later for that part? |
00:21
π
|
JAA |
Articles and news outlinks, sure. |
00:22
π
|
JAA |
(And just to be sure it's not missed, the images are offsite.) |
00:23
π
|
arkiver |
cbistatic yeah |
00:23
π
|
arkiver |
cbsistatic |
00:23
π
|
arkiver |
getting those |
00:24
π
|
JAA |
:-) |
00:30
π
|
arkiver |
tracker is online at gamerankings |
00:30
π
|
arkiver |
we can use some targets :P |
00:32
π
|
JAA |
Kaz, kiska: ^ |
00:34
π
|
arkiver |
alright ready I think |
00:35
π
|
arkiver |
it was an easy site |
00:36
π
|
arkiver |
will do 10 IDs/item |
00:36
π
|
|
Terbium has joined #archiveteam-bs |
00:38
π
|
|
superkuh has joined #archiveteam-bs |
00:44
π
|
arkiver |
JAA: using FOS now |
00:44
π
|
arkiver |
SketchCow: gamerankings is going to FOS |
00:45
π
|
arkiver |
it's on the tracker, probably need the new wget-lua |
00:45
π
|
arkiver |
I got a CONSSLERR with the old wget-lua versions |
00:49
π
|
arkiver |
tiny problem, working on fix |
00:49
π
|
arkiver |
tracker paused |
00:52
π
|
|
odemgi_ has joined #archiveteam-bs |
00:53
π
|
arkiver |
updated and items requeued |
00:54
π
|
|
Terbium has quit IRC (Quit: Page closed) |
00:59
π
|
|
odemgi has quit IRC (Read error: Operation timed out) |
01:00
π
|
arkiver |
updated again |
01:00
π
|
|
Terbium has joined #archiveteam-bs |
01:02
π
|
arkiver |
jodizzle: thanks for staying up to date :P |
01:03
π
|
jodizzle |
Once again, the hero of archiveteam is revealed |
01:04
π
|
arkiver |
haha, :P |
01:04
π
|
arkiver |
well all should be running good now |
01:04
π
|
* |
arkiver is afk for some food |
01:09
π
|
arkiver |
made it the default project |
01:13
π
|
arkiver |
limiting at 2000 items/min |
01:13
π
|
arkiver |
should be done soon |
01:14
π
|
|
odemgi has joined #archiveteam-bs |
01:14
π
|
JAA |
2k per minute, wow. And Fusl isn't even on it. |
01:14
π
|
JAA |
Lots of 0 MB items, is that legit? |
01:14
π
|
arkiver |
yeah |
01:15
π
|
arkiver |
only a few ranges actually have games |
01:15
π
|
JAA |
Oh, yano is on it, nvm. |
01:15
π
|
JAA |
Right. |
01:15
π
|
arkiver |
others just 302 to the main page |
01:15
π
|
yano |
lol |
01:15
π
|
yano |
hai |
01:15
π
|
JAA |
:-) |
01:15
π
|
arkiver |
I'll switch default back to yahoo :P we need it more there |
01:16
π
|
arkiver |
yano: have fun |
01:16
π
|
yano |
:D |
01:16
π
|
JAA |
(And don't tell Fusl. :-P) |
01:16
π
|
arkiver |
heh |
01:16
π
|
arkiver |
I only queued 0-999999 |
01:16
π
|
arkiver |
didn't see any IDs above 999999 |
01:17
π
|
JAA |
Yeah, might be worth checking the files returned over 925k or so if there's anything that isn't tiny. |
01:17
π
|
arkiver |
will just queue 1000000-1499999 after this |
01:17
π
|
arkiver |
should be all 0 MB then |
01:17
π
|
JAA |
Or that. |
01:18
π
|
arkiver |
to be save :P |
01:18
π
|
JAA |
"Because we can." |
01:18
π
|
arkiver |
:P |
01:18
π
|
arkiver |
and website is fast sooo |
01:18
π
|
|
odemgi_ has quit IRC (Read error: Operation timed out) |
01:18
π
|
yano |
i set these up on my phone while i was waiting for my partner to come out of his house; i got a notification is got added to the tracker :D |
01:18
π
|
arkiver |
nice! |
01:19
π
|
* |
arkiver is really off for food now |
01:19
π
|
yano |
i almost got this down to a science :3 |
01:21
π
|
yano |
s/is got added/it got added/ |
01:24
π
|
yano |
i just need to script the git clone of the latest *-grab repo on the GitHub page :3 |
01:28
π
|
|
odemgi_ has joined #archiveteam-bs |
01:36
π
|
|
odemgi has quit IRC (Read error: Operation timed out) |
01:38
π
|
|
d5f4a3622 has joined #archiveteam-bs |
01:41
π
|
|
odemgi has joined #archiveteam-bs |
01:44
π
|
|
atbk_ has joined #archiveteam-bs |
01:47
π
|
|
odemgi_ has quit IRC (Read error: Operation timed out) |
01:48
π
|
|
atbk has quit IRC (Ping timeout: 745 seconds) |
01:52
π
|
|
odemgi_ has joined #archiveteam-bs |
01:56
π
|
|
odemgi has quit IRC (Read error: Operation timed out) |
01:59
π
|
yano |
done |
02:01
π
|
yano |
arkiver: are you gonna queue the next batch for gameranking? |
02:04
π
|
|
odemgi has joined #archiveteam-bs |
02:08
π
|
|
odemgi_ has quit IRC (Read error: Operation timed out) |
02:14
π
|
|
d5f4a3622 has quit IRC (Read error: Connection reset by peer) |
02:25
π
|
|
d5f4a3622 has joined #archiveteam-bs |
02:26
π
|
|
odemgi_ has joined #archiveteam-bs |
02:29
π
|
yano |
i stopped all of my workers gracefully |
02:29
π
|
|
odemgi has quit IRC (Read error: Operation timed out) |
02:38
π
|
|
odemgi has joined #archiveteam-bs |
02:41
π
|
|
godane1 has quit IRC (Read error: Operation timed out) |
02:42
π
|
|
godane has joined #archiveteam-bs |
02:47
π
|
|
odemgi_ has quit IRC (Ping timeout: 610 seconds) |
02:49
π
|
|
odemgi has quit IRC (Read error: Operation timed out) |
02:55
π
|
|
HP_Archiv has joined #archiveteam-bs |
02:57
π
|
|
odemgi has joined #archiveteam-bs |
03:01
π
|
anarcat |
this is crazy https://www.zdnet.com/article/20-vps-providers-to-shut-down-on-monday-giving-customers-two-days-to-save-their-data/ |
03:06
π
|
|
odemgi_ has joined #archiveteam-bs |
03:08
π
|
|
godane has quit IRC (Ping timeout: 745 seconds) |
03:09
π
|
|
odemgi has quit IRC (Ping timeout: 252 seconds) |
03:15
π
|
|
godane has joined #archiveteam-bs |
03:15
π
|
jodizzle |
Hmm. We should probably grab the websites for those services |
03:20
π
|
Ryz |
anarcat, jodizzle - doing it right now~ |
03:20
π
|
Ryz |
Baking up a list - better than having to toss it in one by one |
03:21
π
|
|
odemgi has joined #archiveteam-bs |
03:23
π
|
jodizzle |
Thanks Ryz |
03:23
π
|
|
odemgi_ has quit IRC (Ping timeout: 252 seconds) |
03:30
π
|
Ryz |
If there's more like that anarcat and anyone else that read the article, pitch in more info~ |
03:48
π
|
|
JackT has joined #archiveteam-bs |
03:56
π
|
|
Ravenloft has quit IRC (Read error: Connection reset by peer) |
04:22
π
|
|
odemgi_ has joined #archiveteam-bs |
04:24
π
|
|
odemgi has quit IRC (Ping timeout: 252 seconds) |
04:35
π
|
|
WooWoo289 has joined #archiveteam-bs |
04:55
π
|
|
coldon2dr has joined #archiveteam-bs |
04:58
π
|
|
ryangerma has joined #archiveteam-bs |
04:58
π
|
ryangerma |
hi i have this IBM CD i found but it says US Government users Restrict Rights use duplication or disclosure by GSA ADP Schedule Contract with IBM corp since it wasnt from a government entity it doesnt apply right? Can I send it to you? |
05:00
π
|
|
Terbium has quit IRC (Read error: Operation timed out) |
05:00
π
|
|
qw3rty has joined #archiveteam-bs |
05:01
π
|
astrid |
that means "there are extra rules that apply to you if you are the government" but unless you are government you can do more or less what you want |
05:05
π
|
|
coldon2dr has quit IRC (Leaving) |
05:07
π
|
|
qw3rty2 has quit IRC (Ping timeout: 745 seconds) |
05:07
π
|
|
ryangerma has quit IRC (Ping timeout: 260 seconds) |
05:17
π
|
|
Craigle has quit IRC () |
05:17
π
|
|
WooWoo289 is now known as Craigle |
08:57
π
|
|
britm0b has joined #archiveteam-bs |
08:58
π
|
|
Atom-- has joined #archiveteam-bs |
08:59
π
|
|
yano_ has joined #archiveteam-bs |
08:59
π
|
|
dxrt- has joined #archiveteam-bs |
09:00
π
|
|
Hoolootwo has joined #archiveteam-bs |
09:00
π
|
|
foureyes_ has joined #archiveteam-bs |
09:02
π
|
|
odemgi has joined #archiveteam-bs |
09:02
π
|
|
is-_ has joined #archiveteam-bs |
09:02
π
|
|
Dj-Wawa_ has joined #archiveteam-bs |
09:02
π
|
|
Craigle has quit IRC (Quit: The Lounge - https://thelounge.chat) |
09:02
π
|
|
klg_ has joined #archiveteam-bs |
09:07
π
|
|
ats_ has joined #archiveteam-bs |
09:12
π
|
|
odemgi_ has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
ats has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
i0npulse has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
purplebot has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
kiska has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
Flashfire has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
Atom__ has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
britmob_ has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
britmob has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
pew has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
klg has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
ranma has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
OrIdow6 has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
deevious has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
Dj-Wawa has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
wp494 has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
lempamo has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
VoynichCr has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
dxrt has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
yano has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
coderobe has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
foureyes has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
is- has quit IRC (se.hub irc.underworld.no) |
09:12
π
|
|
Hooloovoo has quit IRC (se.hub irc.underworld.no) |
09:25
π
|
|
Medowar has joined #archiveteam-bs |
09:28
π
|
|
wp494 has joined #archiveteam-bs |
09:28
π
|
|
ranma has joined #archiveteam-bs |
09:28
π
|
|
britmob has joined #archiveteam-bs |
09:29
π
|
|
lempamo has joined #archiveteam-bs |
09:37
π
|
|
dxrt- is now known as dxrt |
09:37
π
|
|
dxrt has quit IRC (ZNC - http://znc.sourceforge.net) |
09:37
π
|
|
dxrt has joined #archiveteam-bs |
09:38
π
|
|
svchfoo3 sets mode: +o dxrt |
09:38
π
|
|
svchfoo1 sets mode: +o dxrt |
09:39
π
|
|
VoynichCr has joined #archiveteam-bs |
09:44
π
|
|
tech234a has quit IRC (Quit: Connection closed for inactivity) |
10:09
π
|
|
OrIdow6 has joined #archiveteam-bs |
10:09
π
|
|
pew has joined #archiveteam-bs |
10:11
π
|
|
i0npulse has joined #archiveteam-bs |
10:24
π
|
|
deevious has joined #archiveteam-bs |
11:14
π
|
Kaz |
from #warrior: |
11:14
π
|
Kaz |
The warrior wikipage doesn't point anyone towards the docker container. This is much easier for a ton of people than downloading the VM, these days |
11:14
π
|
Kaz |
if someone wants to have a go at fixing that, please do |
11:20
π
|
|
VerifiedJ has joined #archiveteam-bs |
11:27
π
|
LowLevelM |
It does |
11:29
π
|
Kaz |
you're right. Could do with being more prominent though |
11:51
π
|
|
kiska has joined #archiveteam-bs |
11:51
π
|
|
svchfoo1 sets mode: +o kiska |
11:51
π
|
|
svchfoo3 sets mode: +o kiska |
12:44
π
|
|
BlueMax has quit IRC (Quit: Leaving) |
12:59
π
|
|
foureyes_ is now known as foureyes |
13:11
π
|
|
indelible has joined #archiveteam-bs |
13:27
π
|
|
yano_ is now known as yano |
13:29
π
|
|
yano has quit IRC (WeeChat, The Better IRC Client, https://weechat.org/) |
13:32
π
|
|
yano has joined #archiveteam-bs |
13:32
π
|
|
i0npulse has quit IRC (Ping timeout: 248 seconds) |
13:32
π
|
|
i0npulse has joined #archiveteam-bs |
14:12
π
|
|
indelible has quit IRC (Remote host closed the connection) |
14:12
π
|
|
indelible has joined #archiveteam-bs |
14:39
π
|
markedL |
We need to start plays.tv for a 1 day grab |
15:07
π
|
jrwr |
@Kaz I've updated the page, I agree as well |
15:08
π
|
Kaz |
Cheers |
15:09
π
|
jrwr |
at some point, I think it would be cool to have something like the alpine virtualbox image auto built for a rPi and require a 64GB SD Card and offer that up for super easy installs |
15:10
π
|
jrwr |
something to throw at the pile :) |
15:11
π
|
jrwr |
throw a X Server and Firefox ESR so you can manage it right from the Pi |
15:11
π
|
anarcat |
Ryz: thanks! |
15:23
π
|
markedL |
I'm not a fan of having RPi's mixed in with servers |
15:26
π
|
jrwr |
for the standard warrior, most people are using Desktops running virtual box |
15:26
π
|
jrwr |
with a rPi4 its faster then some cheap VPSes |
15:29
π
|
|
MaximeleG has joined #archiveteam-bs |
15:32
π
|
|
tech234a has joined #archiveteam-bs |
15:33
π
|
|
SoraUta has joined #archiveteam-bs |
15:37
π
|
|
killsushi has joined #archiveteam-bs |
15:41
π
|
|
DigiDigi has quit IRC (Remote host closed the connection) |
16:27
π
|
|
killsushi has quit IRC (Quit: Leaving) |
16:29
π
|
SketchCow |
Hold up, motherfuckers, I'm uploading 234gb of Professional Darts Competitions to IA |
16:29
π
|
SketchCow |
I mean, who cares about Yahoo |
16:29
π
|
SketchCow |
Daaaaaaarts |
16:38
π
|
|
coderobe has joined #archiveteam-bs |
16:44
π
|
|
phiresky9 has joined #archiveteam-bs |
16:46
π
|
|
phiresky9 is now known as phiresky |
16:50
π
|
|
DigiDigi has joined #archiveteam-bs |
17:02
π
|
SketchCow |
arkiver: As usual, you set up gamerankings so it's thousands of tiny, tiny files, which will take a long time to process. |
17:03
π
|
SketchCow |
I see it's at 7.2gb and already past 16,663 items in one object. |
17:03
π
|
SketchCow |
Try and not do that again, next time |
17:03
π
|
|
X-Scale` has joined #archiveteam-bs |
17:10
π
|
|
X-Scale has quit IRC (Ping timeout: 610 seconds) |
17:10
π
|
|
X-Scale` is now known as X-Scale |
17:18
π
|
|
Craigle has joined #archiveteam-bs |
17:18
π
|
Frogging |
https://arstechnica.com/tech-policy/2019/12/verizon-reportedly-blocks-archivists-from-yahoo-groups-days-before-deletion/ |
17:18
π
|
JAA |
PurpleSym: purplebot's disappeared. |
17:18
π
|
Frogging |
that's wack, yo |
17:19
π
|
jrwr |
Traffic has increased to the wiki, I've done some light mitigations to ensure it survives |
17:21
π
|
JAA |
Inb4 508 Resource Limit Is Reached |
17:22
π
|
|
SoraUta has quit IRC (Ping timeout: 610 seconds) |
17:22
π
|
jrwr |
I do have a special error page for that |
17:22
π
|
jrwr |
it links to the wayback version of the wiki (current latest version) |
17:23
π
|
jrwr |
https://archiveteam.org/508.html |
17:23
π
|
Frogging |
nice |
17:23
π
|
JAA |
Ah right, cool! |
17:24
π
|
jrwr |
now, you can stil bury the webserver, but its super hard to cause the 508 to bury itself |
17:24
π
|
* |
Frogging gets the shovel and starts digging |
17:24
π
|
Frogging |
:p |
17:24
π
|
* |
JAA grabs the qwarc cannon. |
17:28
π
|
jrwr |
Real time usage: 0.114 seconds |
17:28
π
|
jrwr |
so 100ms to gen a page |
17:28
π
|
jrwr |
not too bad |
17:28
π
|
jrwr |
but we have like 10 Chinese bots scraping the site right now |
17:28
π
|
Frogging |
lol |
17:33
π
|
jrwr |
its interesting, I'm watching the access logs to yahoo_groups |
17:34
π
|
jrwr |
most are now coming off twitter and Mastodon |
17:34
π
|
|
purplebot has joined #archiveteam-bs |
17:34
π
|
jrwr |
and the sheer amount of Mastodon instances |
17:35
π
|
markedL |
I wonder if that page is up to date sufficiently |
17:35
π
|
PurpleSym |
JAA: Itβs back. |
18:15
π
|
|
Medowar has quit IRC (Quit: Connection closed for inactivity) |
18:50
π
|
|
systwi_ is now known as systwi |
19:38
π
|
|
jc86035 has joined #archiveteam-bs |
19:54
π
|
|
DogsRNice has joined #archiveteam-bs |
19:58
π
|
|
Jopik has joined #archiveteam-bs |
19:59
π
|
|
Jopik has quit IRC (Client Quit) |
19:59
π
|
|
Jopik has joined #archiveteam-bs |
20:27
π
|
|
MaximeleG has quit IRC (Quit: MaximeleG) |
20:48
π
|
|
tech234a has quit IRC (Quit: Connection closed for inactivity) |
20:50
π
|
|
tech234a has joined #archiveteam-bs |
20:52
π
|
|
HP_Archiv has quit IRC (Read error: Connection reset by peer) |
20:58
π
|
|
jamiew has joined #archiveteam-bs |
21:04
π
|
|
OrIdow6 has quit IRC (Ping timeout: 248 seconds) |
21:28
π
|
|
BlueMax has joined #archiveteam-bs |
21:46
π
|
|
bluefoo has quit IRC (Ping timeout: 255 seconds) |
21:55
π
|
|
X-Scale` has joined #archiveteam-bs |
21:59
π
|
|
X-Scale has quit IRC (Read error: Operation timed out) |
21:59
π
|
|
X-Scale` is now known as X-Scale |
22:02
π
|
|
HP_Archiv has joined #archiveteam-bs |
22:12
π
|
|
jamiew has quit IRC (zzz) |
22:18
π
|
|
bluefoo has joined #archiveteam-bs |
23:00
π
|
|
kode54 has quit IRC (Quit: The Lounge - https://thelounge.chat) |
23:27
π
|
|
britm0b has quit IRC (Read error: Connection reset by peer) |
23:27
π
|
|
britmob has quit IRC (Read error: Connection reset by peer) |
23:31
π
|
|
britmob has joined #archiveteam-bs |
23:31
π
|
|
OrIdow6 has joined #archiveteam-bs |
23:58
π
|
|
jc86035 has quit IRC (Quit: Connection closed for inactivity) |