[00:56] *** IceMaster has joined #archiveteam-bs [00:59] arkiver: That person that that made that tweet about ISP injection that you linked to in #archiveteam awhile ago messaged me on Twitter and said he saw another one, and I think he wants me to pass this link along to you. https://www.dropbox.com/s/uxmebnokwfq2i9k/OptimumOnline-HTMLInjection-Example.zip?dl=0 [01:29] *** hdch has joined #archiveteam-bs [01:59] *** godane has quit IRC (Remote host closed the connection) [02:00] *** godane has joined #archiveteam-bs [02:01] *** VerfiedJ has quit IRC (Quit: Leaving) [02:22] *** DFJustin has quit IRC (Remote host closed the connection) [03:16] *** DFJustin has joined #archiveteam-bs [03:16] *** swebb sets mode: +o DFJustin [03:16] wrt XIVDB, try poking Vekien on Discord? [03:16] "yo can we have a little more time to archive the site" or whatever [03:17] *** DFJustin has quit IRC (Remote host closed the connection) [03:18] *** DFJustin has joined #archiveteam-bs [03:18] *** swebb sets mode: +o DFJustin [03:20] MR9K: If you know where to find them, please try. I'm researching how the site works right now so I can start an ArchiveBot job for it. [03:22] MR9K What discord? [03:22] *** espes__ has quit IRC (Ping timeout: 252 seconds) [03:28] *** tuluu has quit IRC (Read error: Connection refused) [03:29] *** tuluu has joined #archiveteam-bs [03:38] *** K4k_ has joined #archiveteam-bs [03:38] *** K4k_ has quit IRC (Client Quit) [04:02] *** BlueMax has quit IRC (Quit: Leaving) [04:02] *** BlueMax has joined #archiveteam-bs [04:03] *** exoire has quit IRC (Read error: Operation timed out) [04:12] *** espes__ has joined #archiveteam-bs [04:24] Im thinking of making a list for Gopher sites similar to the one for FTP sites [04:25] Flashfire: read the end page [04:25] it specifically mentions a Discord [04:27] *** hdch has quit IRC (Remote host closed the connection) [04:27] *** hdch has joined #archiveteam-bs [04:34] *** qw3rty118 has joined #archiveteam-bs [04:41] *** qw3rty117 has quit IRC (Ping timeout: 600 seconds) [04:42] I'm grabbing the XIVDB with wpull now. [04:42] I created a seed list for the pages I could find (items, places, etc.) except those that have a way too large numeric range to scan in time. [04:43] Let's see how far this gets. [04:43] The site will not be very browsable since much of it is JS-based. [04:43] There's also an API, and I'll look into what can be done with that. [04:44] *** odemg has quit IRC (Ping timeout: 265 seconds) [04:46] JAA This search (with page incremented) seems to get all the items, etc. in their database https://api.xivdb.com/search?language=en&strict=off&page=1 [04:46] Basically, a search with no filters [04:46] *** hdch has quit IRC (Read error: Operation timed out) [04:53] benjins: Thanks. Iterating over that now to see whether my seed list was good. [04:54] JAA since you dont use discord do you want me to get you an API key? [04:56] *** odemg has joined #archiveteam-bs [04:59] Flashfire: Doesn't look like a key is needed. [05:00] "There is no sensitive data on the API and you can only retrieve game related data, nothing regarding members, comments or screenshots." :-/ [05:01] You get more requests per second with an api key [05:02] Also has endpoint restrictions [05:03] Huh, https://github.com/zamnetwork/api mentions nothing about that. [05:03] Is there more API documentation? [05:03] https://xivapi.com/docs [05:04] Uhm, that doesn't look related to xivdb.com. [05:04] It is mentioned in their closing page to go to a discord [05:04] thats what the discord gives me as info [05:05] Huh [05:05] Have questions or fancy making your own database? Get in touch on Discord: https://discord.gg/MFFVHWC. [05:06] Sorry for the slight flood in your DM [05:06] Can you ask them whether XIVAPI is shutting down as well? [05:09] xivapi.com is also linked on the right side on https://xivdb.com/end.html, by the way. [05:09] And described as "A free open source, community driven API for FFXIV". [05:11] XIVAPI is shiny and new they said [05:11] xivapi is replacing xivdb basically [05:11] but only the xivdb api part [05:17] Ah, makes sense. [05:21] xivdb = old site with navigation/search + api [05:21] xivapi = completely new site with only api [05:21] idk what archiveteam is or what you're trying to do but everything in xivdb is outdated and not particularly useful besides for maybe the user comments [05:21] there isnt really anything worth archiving because none of the data is gone [05:21] also, no static content so you can't exactly scrape the site [05:21] the api data in xivapi is the same as xibdb except better schema and actually up to date [05:21] Thats a comment from the devs [05:22] JAA [05:23] Right, I assumed as much. [05:23] Comments and screenshots appear to be unique though. [06:04] *** hdch has joined #archiveteam-bs [06:07] *** wp494 has quit IRC (Read error: Operation timed out) [06:08] *** wp494 has joined #archiveteam-bs [06:47] arkiver: Any update on the UOL Forums error message fix? [08:09] *** IceMaster has quit IRC (Leaving) [08:25] disk fail? https://catalogd.archive.org/history/archiveteam_archivebot_go_20180920180002 [09:04] *** Stilett0 has quit IRC (Read error: Connection reset by peer) [09:04] *** Stiletto has joined #archiveteam-bs [09:05] *** Iridium has quit IRC (Remote host closed the connection) [09:07] *** Iridium has joined #archiveteam-bs [09:28] *** hdch has quit IRC (Quit: oops) [09:30] *** espes__ has quit IRC (Ping timeout: 268 seconds) [09:44] *** BlueMax has quit IRC (Quit: Leaving) [10:42] *** Ganonmast has joined #archiveteam-bs [11:55] *** tomaspark has quit IRC (Read error: Operation timed out) [11:57] *** tomaspark has joined #archiveteam-bs [12:21] *** tomaspark has quit IRC (Ping timeout: 360 seconds) [12:37] *** ubahn has joined #archiveteam-bs [12:42] *** tomaspark has joined #archiveteam-bs [13:39] *** macrosoft has quit IRC (Ping timeout: 633 seconds) [13:54] *** ubahn has quit IRC (Quit: ubahn) [14:09] *** Swiss-- has quit IRC (Read error: Connection reset by peer) [14:11] Oh shit: https://twitter.com/archiveis/status/1081276424781287427 [14:11] [ Please do not use http://archive.IS mirror for linking, use others mirrors [.TODAY .FO .LI .VN .MD .PH]. .IS might stop working soon. ] [14:13] *** Swiss- has joined #archiveteam-bs [14:13] *** ubahn has joined #archiveteam-bs [14:18] Jens: archive.is currently redirects to archive.vn so it appears that it is just an issue with the domain name and not the server (thank god) [14:19] Yea, but it'll cause a lot of link rot if they lose the domain. [14:19] *** Swiss- has quit IRC (Remote host closed the connection) [14:20] *** Swiss- has joined #archiveteam-bs [15:16] *** wp494 has quit IRC (Read error: Operation timed out) [15:16] *** wp494 has joined #archiveteam-bs [15:38] *** ubahn has quit IRC (Quit: ubahn) [15:43] *** ubahn has joined #archiveteam-bs [16:06] *** ubahn has quit IRC (Ping timeout: 260 seconds) [16:10] *** vitzli has joined #archiveteam-bs [16:13] cccccchvbcdtdrbjvfefufigflindueklvhheghvcluk [16:13] wrong chat [16:14] *** Swiss- has quit IRC (Read error: Operation timed out) [16:17] hello yubikey [16:30] *** vitzli has quit IRC (Leaving) [16:58] *** Mateon1 has quit IRC (Ping timeout: 268 seconds) [16:58] *** Mateon1 has joined #archiveteam-bs [17:02] *** chimyatta has joined #archiveteam-bs [17:22] Can anyone give me push access to the uolforums-* repos on GitHub please? Not much time left to restart this... Kaz, astrid, chfoo, DFJustin [17:23] let me try and figure this out [17:23] There's also data on FOS that we need to move out of the way or something. Who has access there besides SketchCow? [17:24] what's your github username JAA [17:25] astrid: JustAnotherArchivist [17:25] ok i made you an owner of the whole github org because i'm too dumb to figure it out [17:25] also you seem trustworthy and motivated [17:26] hope this doesn't backfire _fingers crossed_ [17:28] *** exoire has joined #archiveteam-bs [17:28] Sweet, thanks astrid. :-) [17:32] So any idea about the previous data? [18:03] *** macrosoft has joined #archiveteam-bs [18:04] *** m007a83_ has joined #archiveteam-bs [18:08] *** m007a83 has quit IRC (Ping timeout: 252 seconds) [18:09] *** m007a83_ has quit IRC (Ping timeout: 252 seconds) [18:16] *** macrosoft has quit IRC (Quit: Leaving.) [18:44] So the ArchiveBot job for the Jogos UOL Forums just finished, but it only grabbed 420k URLs, so I'm pretty sure it didn't grab the whole thing. [18:50] New UOL code is ready, just need to sort out the data situation now. [19:16] *** hdch has joined #archiveteam-bs [19:17] *** hdch has quit IRC (Client Quit) [19:17] *** hdch has joined #archiveteam-bs [19:22] *** chimyatta has quit IRC (Ping timeout: 252 seconds) [19:23] *** chimyatta has joined #archiveteam-bs [19:35] *** SmileyG has joined #archiveteam-bs [19:37] *** Smiley has quit IRC (Read error: Operation timed out) [19:55] *** BasDub has joined #archiveteam-bs [19:55] *** DasBub has quit IRC (Read error: Connection reset by peer) [20:17] *** chimyatta has quit IRC (Ping timeout: 252 seconds) [20:18] *** chimyatta has joined #archiveteam-bs [20:28] *** chimyatta has quit IRC (Read error: Connection reset by peer) [20:59] *** madalynn has joined #archiveteam-bs [21:00] *** madalynn has quit IRC (Client Quit) [21:05] JAA: sorry, was absent for some time [21:06] hmm, I thought I had given you access [21:18] arkiver: All good now. Do you have access to FOS to clean up the old data? [21:18] I´ll create a new target an SketchCow can delete the old data later [21:19] Sounds good. [21:19] unfortunately have to go again, back in a few hours [21:19] feel free to update the -grab repo [21:19] * arkiver is back soon [21:20] Yup, I can handle everything except the target. [21:45] *** Oddly has joined #archiveteam-bs [22:30] *** ndiddy has joined #archiveteam-bs [22:37] *** ndiddy has quit IRC () [22:51] *** BlueMax has joined #archiveteam-bs [23:16] *** BlueMax has quit IRC (Quit: Leaving) [23:25] *** phirephly has quit IRC (Quit: ZNC 1.6.3+deb1 - http://znc.in) [23:27] *** phirephly has joined #archiveteam-bs