#archiveteam-bs 2019-12-09,Mon

↑back Search

Time Nickname Message
00:01 πŸ”— arkiver working on it now
00:02 πŸ”— arkiver anyone wants to make a wiki page for it?
00:09 πŸ”— arkiver JAA: shall we just get the review URLs as well :P
00:09 πŸ”— arkiver the outlinks
00:10 πŸ”— Ryz That can work, in a separate job yeah~
00:10 πŸ”— Ryz Ran the 2 jobs with no-offsite-links because of a lot of friction potential
00:11 πŸ”— arkiver will keep it in the same warrior project
00:12 πŸ”— arkiver only getting the HTML, no page requisites
00:13 πŸ”— JAA Yeah, grabbing the reviews as well would be nice. Not sure if it should be done in a second phase though so we're sure we can grab the site itself in time before the shutdown.
00:13 πŸ”— arkiver if we have a large amount of concurrent it should not be a problem
00:13 πŸ”— JAA I know it's only a few million URLs (including the image pages etc.), but still.
00:13 πŸ”— arkiver in case of problem on the outlinks I'll just skip them
00:13 πŸ”— arkiver problems*
00:15 πŸ”— arkiver do we know of any games with a very large number of reviews?
00:15 πŸ”— arkiver possible pagination
00:16 πŸ”— arkiver actually yeah, let's do it in a seperate grab project, we can also get the news URLs then
00:19 πŸ”— JAA Gears of War has 108 reviews and no pagination: https://www.gamerankings.com/xbox360/928234-gears-of-war/articles.html
00:20 πŸ”— JAA But if we just follow any link of https://www.gamerankings.com/PLATFORM/ID-SLUG/ (after detection of PLATFORM and SLUG, for IDs in the item range), any such cases should be covered.
00:21 πŸ”— arkiver yeah, but indeed let's do that later
00:21 πŸ”— arkiver and also get all articles
00:21 πŸ”— arkiver the news
00:21 πŸ”— JAA Why later for that part?
00:21 πŸ”— JAA Articles and news outlinks, sure.
00:22 πŸ”— JAA (And just to be sure it's not missed, the images are offsite.)
00:23 πŸ”— arkiver cbistatic yeah
00:23 πŸ”— arkiver cbsistatic
00:23 πŸ”— arkiver getting those
00:24 πŸ”— JAA :-)
00:30 πŸ”— arkiver tracker is online at gamerankings
00:30 πŸ”— arkiver we can use some targets :P
00:32 πŸ”— JAA Kaz, kiska: ^
00:34 πŸ”— arkiver alright ready I think
00:35 πŸ”— arkiver it was an easy site
00:36 πŸ”— arkiver will do 10 IDs/item
00:36 πŸ”— Terbium has joined #archiveteam-bs
00:38 πŸ”— superkuh has joined #archiveteam-bs
00:44 πŸ”— arkiver JAA: using FOS now
00:44 πŸ”— arkiver SketchCow: gamerankings is going to FOS
00:45 πŸ”— arkiver it's on the tracker, probably need the new wget-lua
00:45 πŸ”— arkiver I got a CONSSLERR with the old wget-lua versions
00:49 πŸ”— arkiver tiny problem, working on fix
00:49 πŸ”— arkiver tracker paused
00:52 πŸ”— odemgi_ has joined #archiveteam-bs
00:53 πŸ”— arkiver updated and items requeued
00:54 πŸ”— Terbium has quit IRC (Quit: Page closed)
00:59 πŸ”— odemgi has quit IRC (Read error: Operation timed out)
01:00 πŸ”— arkiver updated again
01:00 πŸ”— Terbium has joined #archiveteam-bs
01:02 πŸ”— arkiver jodizzle: thanks for staying up to date :P
01:03 πŸ”— jodizzle Once again, the hero of archiveteam is revealed
01:04 πŸ”— arkiver haha, :P
01:04 πŸ”— arkiver well all should be running good now
01:04 πŸ”— * arkiver is afk for some food
01:09 πŸ”— arkiver made it the default project
01:13 πŸ”— arkiver limiting at 2000 items/min
01:13 πŸ”— arkiver should be done soon
01:14 πŸ”— odemgi has joined #archiveteam-bs
01:14 πŸ”— JAA 2k per minute, wow. And Fusl isn't even on it.
01:14 πŸ”— JAA Lots of 0 MB items, is that legit?
01:14 πŸ”— arkiver yeah
01:15 πŸ”— arkiver only a few ranges actually have games
01:15 πŸ”— JAA Oh, yano is on it, nvm.
01:15 πŸ”— JAA Right.
01:15 πŸ”— arkiver others just 302 to the main page
01:15 πŸ”— yano lol
01:15 πŸ”— yano hai
01:15 πŸ”— JAA :-)
01:15 πŸ”— arkiver I'll switch default back to yahoo :P we need it more there
01:16 πŸ”— arkiver yano: have fun
01:16 πŸ”— yano :D
01:16 πŸ”— JAA (And don't tell Fusl. :-P)
01:16 πŸ”— arkiver heh
01:16 πŸ”— arkiver I only queued 0-999999
01:16 πŸ”— arkiver didn't see any IDs above 999999
01:17 πŸ”— JAA Yeah, might be worth checking the files returned over 925k or so if there's anything that isn't tiny.
01:17 πŸ”— arkiver will just queue 1000000-1499999 after this
01:17 πŸ”— arkiver should be all 0 MB then
01:17 πŸ”— JAA Or that.
01:18 πŸ”— arkiver to be save :P
01:18 πŸ”— JAA "Because we can."
01:18 πŸ”— arkiver :P
01:18 πŸ”— arkiver and website is fast sooo
01:18 πŸ”— odemgi_ has quit IRC (Read error: Operation timed out)
01:18 πŸ”— yano i set these up on my phone while i was waiting for my partner to come out of his house; i got a notification is got added to the tracker :D
01:18 πŸ”— arkiver nice!
01:19 πŸ”— * arkiver is really off for food now
01:19 πŸ”— yano i almost got this down to a science :3
01:21 πŸ”— yano s/is got added/it got added/
01:24 πŸ”— yano i just need to script the git clone of the latest *-grab repo on the GitHub page :3
01:28 πŸ”— odemgi_ has joined #archiveteam-bs
01:36 πŸ”— odemgi has quit IRC (Read error: Operation timed out)
01:38 πŸ”— d5f4a3622 has joined #archiveteam-bs
01:41 πŸ”— odemgi has joined #archiveteam-bs
01:44 πŸ”— atbk_ has joined #archiveteam-bs
01:47 πŸ”— odemgi_ has quit IRC (Read error: Operation timed out)
01:48 πŸ”— atbk has quit IRC (Ping timeout: 745 seconds)
01:52 πŸ”— odemgi_ has joined #archiveteam-bs
01:56 πŸ”— odemgi has quit IRC (Read error: Operation timed out)
01:59 πŸ”— yano done
02:01 πŸ”— yano arkiver: are you gonna queue the next batch for gameranking?
02:04 πŸ”— odemgi has joined #archiveteam-bs
02:08 πŸ”— odemgi_ has quit IRC (Read error: Operation timed out)
02:14 πŸ”— d5f4a3622 has quit IRC (Read error: Connection reset by peer)
02:25 πŸ”— d5f4a3622 has joined #archiveteam-bs
02:26 πŸ”— odemgi_ has joined #archiveteam-bs
02:29 πŸ”— yano i stopped all of my workers gracefully
02:29 πŸ”— odemgi has quit IRC (Read error: Operation timed out)
02:38 πŸ”— odemgi has joined #archiveteam-bs
02:41 πŸ”— godane1 has quit IRC (Read error: Operation timed out)
02:42 πŸ”— godane has joined #archiveteam-bs
02:47 πŸ”— odemgi_ has quit IRC (Ping timeout: 610 seconds)
02:49 πŸ”— odemgi has quit IRC (Read error: Operation timed out)
02:55 πŸ”— HP_Archiv has joined #archiveteam-bs
02:57 πŸ”— odemgi has joined #archiveteam-bs
03:01 πŸ”— anarcat this is crazy https://www.zdnet.com/article/20-vps-providers-to-shut-down-on-monday-giving-customers-two-days-to-save-their-data/
03:06 πŸ”— odemgi_ has joined #archiveteam-bs
03:08 πŸ”— godane has quit IRC (Ping timeout: 745 seconds)
03:09 πŸ”— odemgi has quit IRC (Ping timeout: 252 seconds)
03:15 πŸ”— godane has joined #archiveteam-bs
03:15 πŸ”— jodizzle Hmm. We should probably grab the websites for those services
03:20 πŸ”— Ryz anarcat, jodizzle - doing it right now~
03:20 πŸ”— Ryz Baking up a list - better than having to toss it in one by one
03:21 πŸ”— odemgi has joined #archiveteam-bs
03:23 πŸ”— jodizzle Thanks Ryz
03:23 πŸ”— odemgi_ has quit IRC (Ping timeout: 252 seconds)
03:30 πŸ”— Ryz If there's more like that anarcat and anyone else that read the article, pitch in more info~
03:48 πŸ”— JackT has joined #archiveteam-bs
03:56 πŸ”— Ravenloft has quit IRC (Read error: Connection reset by peer)
04:22 πŸ”— odemgi_ has joined #archiveteam-bs
04:24 πŸ”— odemgi has quit IRC (Ping timeout: 252 seconds)
04:35 πŸ”— WooWoo289 has joined #archiveteam-bs
04:55 πŸ”— coldon2dr has joined #archiveteam-bs
04:58 πŸ”— ryangerma has joined #archiveteam-bs
04:58 πŸ”— ryangerma hi i have this IBM CD i found but it says US Government users Restrict Rights use duplication or disclosure by GSA ADP Schedule Contract with IBM corp since it wasnt from a government entity it doesnt apply right? Can I send it to you?
05:00 πŸ”— Terbium has quit IRC (Read error: Operation timed out)
05:00 πŸ”— qw3rty has joined #archiveteam-bs
05:01 πŸ”— astrid that means "there are extra rules that apply to you if you are the government" but unless you are government you can do more or less what you want
05:05 πŸ”— coldon2dr has quit IRC (Leaving)
05:07 πŸ”— qw3rty2 has quit IRC (Ping timeout: 745 seconds)
05:07 πŸ”— ryangerma has quit IRC (Ping timeout: 260 seconds)
05:17 πŸ”— Craigle has quit IRC ()
05:17 πŸ”— WooWoo289 is now known as Craigle
08:57 πŸ”— britm0b has joined #archiveteam-bs
08:58 πŸ”— Atom-- has joined #archiveteam-bs
08:59 πŸ”— yano_ has joined #archiveteam-bs
08:59 πŸ”— dxrt- has joined #archiveteam-bs
09:00 πŸ”— Hoolootwo has joined #archiveteam-bs
09:00 πŸ”— foureyes_ has joined #archiveteam-bs
09:02 πŸ”— odemgi has joined #archiveteam-bs
09:02 πŸ”— is-_ has joined #archiveteam-bs
09:02 πŸ”— Dj-Wawa_ has joined #archiveteam-bs
09:02 πŸ”— Craigle has quit IRC (Quit: The Lounge - https://thelounge.chat)
09:02 πŸ”— klg_ has joined #archiveteam-bs
09:07 πŸ”— ats_ has joined #archiveteam-bs
09:12 πŸ”— odemgi_ has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— ats has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— i0npulse has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— purplebot has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— kiska has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— Flashfire has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— Atom__ has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— britmob_ has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— britmob has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— pew has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— klg has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— ranma has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— OrIdow6 has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— deevious has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— Dj-Wawa has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— wp494 has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— lempamo has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— VoynichCr has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— dxrt has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— yano has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— coderobe has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— foureyes has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— is- has quit IRC (se.hub irc.underworld.no)
09:12 πŸ”— Hooloovoo has quit IRC (se.hub irc.underworld.no)
09:25 πŸ”— Medowar has joined #archiveteam-bs
09:28 πŸ”— wp494 has joined #archiveteam-bs
09:28 πŸ”— ranma has joined #archiveteam-bs
09:28 πŸ”— britmob has joined #archiveteam-bs
09:29 πŸ”— lempamo has joined #archiveteam-bs
09:37 πŸ”— dxrt- is now known as dxrt
09:37 πŸ”— dxrt has quit IRC (ZNC - http://znc.sourceforge.net)
09:37 πŸ”— dxrt has joined #archiveteam-bs
09:38 πŸ”— svchfoo3 sets mode: +o dxrt
09:38 πŸ”— svchfoo1 sets mode: +o dxrt
09:39 πŸ”— VoynichCr has joined #archiveteam-bs
09:44 πŸ”— tech234a has quit IRC (Quit: Connection closed for inactivity)
10:09 πŸ”— OrIdow6 has joined #archiveteam-bs
10:09 πŸ”— pew has joined #archiveteam-bs
10:11 πŸ”— i0npulse has joined #archiveteam-bs
10:24 πŸ”— deevious has joined #archiveteam-bs
11:14 πŸ”— Kaz from #warrior:
11:14 πŸ”— Kaz The warrior wikipage doesn't point anyone towards the docker container. This is much easier for a ton of people than downloading the VM, these days
11:14 πŸ”— Kaz if someone wants to have a go at fixing that, please do
11:20 πŸ”— VerifiedJ has joined #archiveteam-bs
11:27 πŸ”— LowLevelM It does
11:29 πŸ”— Kaz you're right. Could do with being more prominent though
11:51 πŸ”— kiska has joined #archiveteam-bs
11:51 πŸ”— svchfoo1 sets mode: +o kiska
11:51 πŸ”— svchfoo3 sets mode: +o kiska
12:44 πŸ”— BlueMax has quit IRC (Quit: Leaving)
12:59 πŸ”— foureyes_ is now known as foureyes
13:11 πŸ”— indelible has joined #archiveteam-bs
13:27 πŸ”— yano_ is now known as yano
13:29 πŸ”— yano has quit IRC (WeeChat, The Better IRC Client, https://weechat.org/)
13:32 πŸ”— yano has joined #archiveteam-bs
13:32 πŸ”— i0npulse has quit IRC (Ping timeout: 248 seconds)
13:32 πŸ”— i0npulse has joined #archiveteam-bs
14:12 πŸ”— indelible has quit IRC (Remote host closed the connection)
14:12 πŸ”— indelible has joined #archiveteam-bs
14:39 πŸ”— markedL We need to start plays.tv for a 1 day grab
15:07 πŸ”— jrwr @Kaz I've updated the page, I agree as well
15:08 πŸ”— Kaz Cheers
15:09 πŸ”— jrwr at some point, I think it would be cool to have something like the alpine virtualbox image auto built for a rPi and require a 64GB SD Card and offer that up for super easy installs
15:10 πŸ”— jrwr something to throw at the pile :)
15:11 πŸ”— jrwr throw a X Server and Firefox ESR so you can manage it right from the Pi
15:11 πŸ”— anarcat Ryz: thanks!
15:23 πŸ”— markedL I'm not a fan of having RPi's mixed in with servers
15:26 πŸ”— jrwr for the standard warrior, most people are using Desktops running virtual box
15:26 πŸ”— jrwr with a rPi4 its faster then some cheap VPSes
15:29 πŸ”— MaximeleG has joined #archiveteam-bs
15:32 πŸ”— tech234a has joined #archiveteam-bs
15:33 πŸ”— SoraUta has joined #archiveteam-bs
15:37 πŸ”— killsushi has joined #archiveteam-bs
15:41 πŸ”— DigiDigi has quit IRC (Remote host closed the connection)
16:27 πŸ”— killsushi has quit IRC (Quit: Leaving)
16:29 πŸ”— SketchCow Hold up, motherfuckers, I'm uploading 234gb of Professional Darts Competitions to IA
16:29 πŸ”— SketchCow I mean, who cares about Yahoo
16:29 πŸ”— SketchCow Daaaaaaarts
16:38 πŸ”— coderobe has joined #archiveteam-bs
16:44 πŸ”— phiresky9 has joined #archiveteam-bs
16:46 πŸ”— phiresky9 is now known as phiresky
16:50 πŸ”— DigiDigi has joined #archiveteam-bs
17:02 πŸ”— SketchCow arkiver: As usual, you set up gamerankings so it's thousands of tiny, tiny files, which will take a long time to process.
17:03 πŸ”— SketchCow I see it's at 7.2gb and already past 16,663 items in one object.
17:03 πŸ”— SketchCow Try and not do that again, next time
17:03 πŸ”— X-Scale` has joined #archiveteam-bs
17:10 πŸ”— X-Scale has quit IRC (Ping timeout: 610 seconds)
17:10 πŸ”— X-Scale` is now known as X-Scale
17:18 πŸ”— Craigle has joined #archiveteam-bs
17:18 πŸ”— Frogging https://arstechnica.com/tech-policy/2019/12/verizon-reportedly-blocks-archivists-from-yahoo-groups-days-before-deletion/
17:18 πŸ”— JAA PurpleSym: purplebot's disappeared.
17:18 πŸ”— Frogging that's wack, yo
17:19 πŸ”— jrwr Traffic has increased to the wiki, I've done some light mitigations to ensure it survives
17:21 πŸ”— JAA Inb4 508 Resource Limit Is Reached
17:22 πŸ”— SoraUta has quit IRC (Ping timeout: 610 seconds)
17:22 πŸ”— jrwr I do have a special error page for that
17:22 πŸ”— jrwr it links to the wayback version of the wiki (current latest version)
17:23 πŸ”— jrwr https://archiveteam.org/508.html
17:23 πŸ”— Frogging nice
17:23 πŸ”— JAA Ah right, cool!
17:24 πŸ”— jrwr now, you can stil bury the webserver, but its super hard to cause the 508 to bury itself
17:24 πŸ”— * Frogging gets the shovel and starts digging
17:24 πŸ”— Frogging :p
17:24 πŸ”— * JAA grabs the qwarc cannon.
17:28 πŸ”— jrwr Real time usage: 0.114 seconds
17:28 πŸ”— jrwr so 100ms to gen a page
17:28 πŸ”— jrwr not too bad
17:28 πŸ”— jrwr but we have like 10 Chinese bots scraping the site right now
17:28 πŸ”— Frogging lol
17:33 πŸ”— jrwr its interesting, I'm watching the access logs to yahoo_groups
17:34 πŸ”— jrwr most are now coming off twitter and Mastodon
17:34 πŸ”— purplebot has joined #archiveteam-bs
17:34 πŸ”— jrwr and the sheer amount of Mastodon instances
17:35 πŸ”— markedL I wonder if that page is up to date sufficiently
17:35 πŸ”— PurpleSym JAA: It’s back.
18:15 πŸ”— Medowar has quit IRC (Quit: Connection closed for inactivity)
18:50 πŸ”— systwi_ is now known as systwi
19:38 πŸ”— jc86035 has joined #archiveteam-bs
19:54 πŸ”— DogsRNice has joined #archiveteam-bs
19:58 πŸ”— Jopik has joined #archiveteam-bs
19:59 πŸ”— Jopik has quit IRC (Client Quit)
19:59 πŸ”— Jopik has joined #archiveteam-bs
20:27 πŸ”— MaximeleG has quit IRC (Quit: MaximeleG)
20:48 πŸ”— tech234a has quit IRC (Quit: Connection closed for inactivity)
20:50 πŸ”— tech234a has joined #archiveteam-bs
20:52 πŸ”— HP_Archiv has quit IRC (Read error: Connection reset by peer)
20:58 πŸ”— jamiew has joined #archiveteam-bs
21:04 πŸ”— OrIdow6 has quit IRC (Ping timeout: 248 seconds)
21:28 πŸ”— BlueMax has joined #archiveteam-bs
21:46 πŸ”— bluefoo has quit IRC (Ping timeout: 255 seconds)
21:55 πŸ”— X-Scale` has joined #archiveteam-bs
21:59 πŸ”— X-Scale has quit IRC (Read error: Operation timed out)
21:59 πŸ”— X-Scale` is now known as X-Scale
22:02 πŸ”— HP_Archiv has joined #archiveteam-bs
22:12 πŸ”— jamiew has quit IRC (zzz)
22:18 πŸ”— bluefoo has joined #archiveteam-bs
23:00 πŸ”— kode54 has quit IRC (Quit: The Lounge - https://thelounge.chat)
23:27 πŸ”— britm0b has quit IRC (Read error: Connection reset by peer)
23:27 πŸ”— britmob has quit IRC (Read error: Connection reset by peer)
23:31 πŸ”— britmob has joined #archiveteam-bs
23:31 πŸ”— OrIdow6 has joined #archiveteam-bs
23:58 πŸ”— jc86035 has quit IRC (Quit: Connection closed for inactivity)

irclogger-viewer