[00:04] *** robbierut has quit IRC (Read error: Operation timed out) [00:05] that's my screwup [00:05] They'll all be got. [00:10] *** omarroth has joined #archiveteam-bs [00:21] *** marked has quit IRC (Quit: WeeChat 2.2) [00:25] *** marked has joined #archiveteam-bs [00:26] (Continuing from -ot regarding ideas for next projects) [00:26] *** LowLevelM has joined #archiveteam-bs [00:26] phiresky: Yes, Google+ will be (is partially already) in the WBM. Scrolling won't work though because it uses POST requests and the WBM can't handle those. [00:28] VADemon: JamiiForums is also XenForo and those scripts are pretty much ready. [00:28] *** nyany has joined #archiveteam-bs [00:28] We should have done Jamii a while ago but we never came to a decision about member only areas [00:28] Nice. So 1) JamiiForums 2) Bukkit.org (unknown future) 3) Hardforum (will stay afloat) [00:29] 0) or also 1) Reddit due to all the recent bans etc. [00:31] Here is the process for archiving a Thingiverse thing: Get JSON for id, get zip file for id, get json list of photos, get photos [00:31] If it is a work in progress, maybe add it to a list to re-archive [00:32] JAA: is there a way to make a "custom ui" in the internet archive (like those old game VMs) so e.g. reddit can be implemented as a dynamic frontend to json dumps instead of just millions of copies of the almost same website over and over? [00:33] (like https://snew.notabug.io/r/all [00:33] ) [00:35] just asking because i wondered the same thing about the google+ project [00:37] phiresky: It's certainly possible, but I doubt the IA people have the time/resources to develop something specifically for that. But maybe something could be hacked together like that JS search thingy for an old AT project that someone wrote a long time ago. [00:38] yeah i mean like a way for third parties (us) to program UIs against their backing storage [00:40] like these https://archive.org/details/msdos_Oregon_Trail_The_1990 [00:41] This is what I meant: https://archive.org/download/webshots-freeze-frame-index/index.html [00:41] (The download links are broken in this case, but it could e.g. embed a WBM frame or something like that instead.) [00:45] *** marked has quit IRC (Read error: Operation timed out) [00:45] yeah exactly.. was that done with active support of IA or completely independently? [00:49] *** marked has joined #archiveteam-bs [00:49] ok looks like all IA content is directly downloadable and also has CORS headers set [00:50] *** hendi__ has joined #archiveteam-bs [00:51] No idea. I wasn't around yet when that project happened. [00:53] *** hendi_ has quit IRC (Ping timeout: 252 seconds) [00:53] yeah it seems to use http range requests to "abuse" the IA storage to make a kind of database of existing data for the search [00:56] i've done something similar before, only problem is that the data has to be on the server in a way that makes range requests possible (so no to most compressions) [00:57] ah yeah and also you can't really do aggregations quickly, so the comments of a post have to be stored near the post as opposed to the way pushshift.io stores it [00:58] *** killsushi has joined #archiveteam-bs [01:00] *** Evie has joined #archiveteam-bs [01:01] i.e. making a ui for this reddit dump that's already on IA (2005-2017) is not really possible https://archive.org/details/academictorrents_85a5bd50e4c365f8df70240ffd4ecc7dec59912b [01:02] Yeah, that's impossible. [01:03] Maybe we can also grab the API data in our Reddit project and then access that directly. [01:03] Alternatively, we'd have to decompress the Pushshift data and store it differently. [01:04] but since now reddit posts are immutable after 6 months anyways, it would be easy to convert that data to just huge json files per post and make a ui for that [01:04] After 6 months of no activity, I believe. So it's a bit messier, but yeah, that should work. [01:05] mh i thought 6 months fixed cause i've previously tried to comment on stuff that still seemed pretty alive [01:05] all that data is also on google bigquery btw [01:05] so it's also possible to write SQL queries against the whole data fairly easily [01:05] We'll anyway regrab threads until they get archived as part of the project, I think. So it's easy to add some detection of archived threads so they can be processed accordingly. [01:06] I'm pretty sure I've commented on a 2-year-old thread before. [01:06] Get ready for google+ flashover [01:06] But this is getting pretty Reddit-specific, so maybe we should move to the project channel, #shreddit [01:07] can't even find any official information on archiving [01:15] Thanks, JAA! Did the rate limiting coincide with it, or when did that start happening? [01:16] ephemer0l: Uh, what are you referring to? [01:17] That wiki pasted? [01:17] https://twitter.com/textfiles/status/1112494767601053696 [01:18] Ah, I thought you meant something about that wiki is getting rate-limited. [01:18] I don't know what's going on at Google+, wasn't involved much in that project. #googleminus [01:18] sorry, nope [01:18] Thanks :-) [01:19] lol @ the first reply on that tweet [01:21] god dammit [01:23] Who has the first reply? [01:24] https://twitter.com/ChrisEineke/status/1112512726310383616 [01:24] Imma just leave that ther [01:30] *** LowLevelM has left [01:50] *** robbierut has joined #archiveteam-bs [02:00] Flashfire: I think he means limiting it to fetching a single page per minute per client [02:00] but that would be too slow and require way too many clients [02:01] nah [02:01] I think he's just unaware of what we're doing [02:01] more likely that [02:07] *** Exairnous has joined #archiveteam-bs [02:27] *** ndiddy has quit IRC () [02:46] *** ephemer0l has quit IRC (Quit: http://quassel-irc.org - Chat comfortably. Anywhere.) [02:58] *** Despatche has quit IRC (Quit: Read error: Connection reset by deer) [03:09] *** DustinV has joined #archiveteam-bs [03:20] *** omarroth has quit IRC (Read error: Connection reset by peer) [03:33] *** qw3rty115 has joined #archiveteam-bs [03:36] *** qw3rty114 has quit IRC (Ping timeout: 600 seconds) [03:46] *** IanR has joined #archiveteam-bs [03:54] *** odemgi_ has joined #archiveteam-bs [03:56] *** odemgi has quit IRC (Read error: Operation timed out) [04:02] *** Stiletto has quit IRC () [04:03] *** odemg has quit IRC (Ping timeout: 615 seconds) [04:04] *** ephemer0l has joined #archiveteam-bs [04:08] *** Stiletto has joined #archiveteam-bs [04:09] *** odemg has joined #archiveteam-bs [04:22] *** DustinV has quit IRC (Read error: Connection reset by peer) [04:32] *** m007a83_ is now known as m007a83 [04:50] *** dhyan_nat has joined #archiveteam-bs [05:08] *** DustinV has joined #archiveteam-bs [05:36] *** robbierut has quit IRC (Read error: Connection reset by peer) [06:04] *** jut has quit IRC (Ping timeout: 252 seconds) [06:05] *** icedice has quit IRC (Quit: Leaving) [06:11] *** jut has joined #archiveteam-bs [06:29] *** BlueMax has quit IRC (Quit: Leaving) [06:34] *** Exairnous has quit IRC (Read error: Operation timed out) [06:45] *** deevious has joined #archiveteam-bs [06:59] *** julientm has joined #archiveteam-bs [07:07] *** BlueMax has joined #archiveteam-bs [07:16] *** robbierut has joined #archiveteam-bs [08:06] *** Joseph__ has joined #archiveteam-bs [08:07] *** VerifiedJ has quit IRC (Read error: Connection reset by peer) [08:17] *** julientm has quit IRC (Remote host closed the connection) [08:32] *** DustinVF has joined #archiveteam-bs [08:32] *** julientm has joined #archiveteam-bs [08:33] *** DustinV has quit IRC (Ping timeout: 252 seconds) [08:34] *** julientm has quit IRC (Remote host closed the connection) [08:36] *** DustinVF has quit IRC (Read error: Operation timed out) [08:37] *** julientm has joined #archiveteam-bs [08:39] *** IanR has quit IRC (Read error: Connection reset by peer) [08:39] *** DustinV has joined #archiveteam-bs [08:40] *** julientm has quit IRC (Remote host closed the connection) [08:40] *** IanR has joined #archiveteam-bs [08:40] *** julientm has joined #archiveteam-bs [08:56] *** IanR has quit IRC (Read error: Connection reset by peer) [08:57] *** IanR has joined #archiveteam-bs [08:58] *** julientm has quit IRC (Remote host closed the connection) [08:59] *** julientm has joined #archiveteam-bs [09:09] *** DFJustin has quit IRC (Ping timeout: 615 seconds) [09:12] *** julientm has quit IRC (Remote host closed the connection) [09:12] *** julientm has joined #archiveteam-bs [09:14] *** DFJustin has joined #archiveteam-bs [09:19] *** MR9K has quit IRC (Read error: Connection reset by peer) [09:21] *** MR9K has joined #archiveteam-bs [09:47] *** ryry has quit IRC (Ping timeout: 260 seconds) [10:00] *** julientm has quit IRC (Remote host closed the connection) [10:11] *** julientm has joined #archiveteam-bs [10:11] *** BlueMax has quit IRC (Quit: Leaving) [10:25] *** jesso has joined #archiveteam-bs [10:44] *** Oddly has joined #archiveteam-bs [10:58] *** hendi__ has quit IRC (Read error: Connection reset by peer) [11:00] *** hendi has joined #archiveteam-bs [11:07] *** jesso has quit IRC (Quit: jesso) [11:10] *** jesso has joined #archiveteam-bs [11:31] *** killsushi has quit IRC (Quit: Leaving) [11:46] *** dhyan_nat has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** Smiley has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** Mateon1 has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** SketchCow has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** overflowe has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** ats has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** betamax has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** noirscape has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** argus has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** asie has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** Tenebrae has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** K4k__ has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** colona has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** synm0nger has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** MrRadar2 has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** bsmith093 has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** Coderjo has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** BnAboyZ has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** Ganonmast has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** kisspunch has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** Frogging has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** jodizzle has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** VoynichCr has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** odemgi_ has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** t2t2 has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** Atom-- has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** wp494 has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** Hintswen has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** Lord_Nigh has quit IRC (hub.efnet.us irc.efnet.nl) [11:46] *** halt has quit IRC (hub.efnet.us irc.efnet.nl) [11:53] *** dhyan_nat has joined #archiveteam-bs [11:53] *** odemgi_ has joined #archiveteam-bs [11:53] *** Smiley has joined #archiveteam-bs [11:53] *** t2t2 has joined #archiveteam-bs [11:53] *** Mateon1 has joined #archiveteam-bs [11:53] *** Atom-- has joined #archiveteam-bs [11:53] *** wp494 has joined #archiveteam-bs [11:53] *** Hintswen has joined #archiveteam-bs [11:53] *** Lord_Nigh has joined #archiveteam-bs [11:53] *** SketchCow has joined #archiveteam-bs [11:53] *** halt has joined #archiveteam-bs [11:53] *** overflowe has joined #archiveteam-bs [11:53] *** ats has joined #archiveteam-bs [11:53] *** betamax has joined #archiveteam-bs [11:53] *** noirscape has joined #archiveteam-bs [11:53] *** argus has joined #archiveteam-bs [11:53] *** asie has joined #archiveteam-bs [11:53] *** Tenebrae has joined #archiveteam-bs [11:53] *** K4k__ has joined #archiveteam-bs [11:53] *** colona has joined #archiveteam-bs [11:53] *** synm0nger has joined #archiveteam-bs [11:53] *** MrRadar2 has joined #archiveteam-bs [11:53] *** bsmith093 has joined #archiveteam-bs [11:53] *** Coderjo has joined #archiveteam-bs [11:53] *** BnAboyZ has joined #archiveteam-bs [11:53] *** Ganonmast has joined #archiveteam-bs [11:53] *** kisspunch has joined #archiveteam-bs [11:53] *** Frogging has joined #archiveteam-bs [11:53] *** jodizzle has joined #archiveteam-bs [11:53] *** VoynichCr has joined #archiveteam-bs [11:53] *** irc.efnet.nl sets mode: +o MrRadar2 [11:53] *** Fusl sets mode: +o SketchCow [12:26] *** julientm has quit IRC (Read error: Connection reset by peer) [12:34] *** julientm_ has joined #archiveteam-bs [12:38] *** delightfu has joined #archiveteam-bs [12:38] *** delightfu has left [12:39] *** Despatche has joined #archiveteam-bs [12:48] *** julientm_ has quit IRC (Remote host closed the connection) [12:48] *** julientm_ has joined #archiveteam-bs [12:58] *** delightfu has joined #archiveteam-bs [13:18] *** synm0nger has quit IRC (Quit: Wait, what?) [13:23] *** SynMonger has joined #archiveteam-bs [13:27] *** Wizzito has joined #archiveteam-bs [13:50] *** robbierut has quit IRC (Read error: Operation timed out) [13:50] *** robbierut has joined #archiveteam-bs [13:57] *** omarroth has joined #archiveteam-bs [14:00] *** julientm_ has quit IRC (Read error: Connection reset by peer) [14:04] *** julientm has joined #archiveteam-bs [14:09] *** robbierut has quit IRC (Ping timeout: 360 seconds) [14:09] *** robbierut has joined #archiveteam-bs [14:12] *** robbierut has quit IRC (Read error: Connection reset by peer) [14:13] *** robbierut has joined #archiveteam-bs [14:18] *** DustinVF has joined #archiveteam-bs [14:22] *** DustinVFP has joined #archiveteam-bs [14:22] *** deevious has quit IRC (Quit: deevious) [14:22] *** DustinVFP is now known as otherDust [14:23] *** DustinVF has quit IRC (Read error: Operation timed out) [14:27] *** DustinV has quit IRC (Read error: Operation timed out) [14:27] *** otherDust is now known as DustinV [14:33] *** deevious has joined #archiveteam-bs [14:40] *** Wizzito has quit IRC (Quit: Leaving) [14:50] *** DustinV has quit IRC (Remote host closed the connection) [14:51] *** DustinV has joined #archiveteam-bs [14:51] *** DustinV has quit IRC (Read error: Connection reset by peer) [14:52] *** DustinV has joined #archiveteam-bs [15:31] *** delightfu has quit IRC (Quit: http://www.mibbit.com ajax IRC Client) [15:36] *** julientm has quit IRC (Ping timeout: 252 seconds) [15:40] *** julientm has joined #archiveteam-bs [15:44] *** julientm has quit IRC (Remote host closed the connection) [15:45] *** julientm has joined #archiveteam-bs [16:14] *** Dj-Wawa has joined #archiveteam-bs [16:23] *** dhyan_nat has quit IRC (Read error: Operation timed out) [16:27] *** robbierut has quit IRC (Read error: Operation timed out) [16:27] *** robbierut has joined #archiveteam-bs [16:32] *** adinbied has quit IRC (Quit: Leaving) [16:32] *** adinbied has joined #archiveteam-bs [16:36] *** omarroth has quit IRC (Ping timeout: 268 seconds) [16:54] *** Joseph__ has quit IRC (Read error: Connection reset by peer) [16:55] *** VerifiedJ has joined #archiveteam-bs [17:13] *** bsmith093 has quit IRC (Quit: Leaving.) [17:17] *** bsmith093 has joined #archiveteam-bs [17:49] *** robbierut has quit IRC (Read error: Connection reset by peer) [17:50] *** robbierut has joined #archiveteam-bs [17:56] *** Oddly has quit IRC (Ping timeout: 257 seconds) [18:05] *** coderobe has joined #archiveteam-bs [18:20] *** marked has quit IRC (Read error: Operation timed out) [18:22] *** Oddly has joined #archiveteam-bs [18:22] *** Exairnous has joined #archiveteam-bs [18:25] *** marked has joined #archiveteam-bs [18:47] *** robbierut has quit IRC (Read error: Connection reset by peer) [18:47] *** robbierut has joined #archiveteam-bs [18:50] *** Exairnous has quit IRC (Remote host closed the connection) [18:51] *** Exairnous has joined #archiveteam-bs [18:52] *** icedice has joined #archiveteam-bs [18:56] *** Oddly has quit IRC (Ping timeout: 255 seconds) [18:56] wow, hiddenpalace just nuked all the nintendo-based prototypes they hosted. rip protos [18:57] http://hiddenpalace.org/w/index.php?title=Template%3ADoNotUploadList&action=historysubmit&type=revision&diff=25459&oldid=14882 [18:58] *** bsmith093 has quit IRC (Quit: Leaving.) [18:59] "24 year old security researcher Zammis Clark has pleaded guilty for hacking into Nintendo and Microsoft servers to gain access to confidential information." ... this may explain why. [19:00] *** robbierut has quit IRC (Read error: Connection reset by peer) [19:02] *** Odd0002_ has joined #archiveteam-bs [19:02] *** julientm has quit IRC (Read error: Connection reset by peer) [19:02] *** robbierut has joined #archiveteam-bs [19:02] looks like IA did NOT have a mirror of hiddenpalace either [19:03] *** Despatche has quit IRC (Read error: Operation timed out) [19:03] hmm i stand corrected, it looks like most of what was removed was covered [19:04] *** Exairnous has quit IRC (Read error: Operation timed out) [19:06] *** Odd0002 has quit IRC (Ping timeout: 615 seconds) [19:06] *** Odd0002_ is now known as Odd0002 [19:12] *** Exairnous has joined #archiveteam-bs [19:17] *** dhyan_nat has joined #archiveteam-bs [19:22] *** Exairnous has quit IRC (Ping timeout: 615 seconds) [19:29] *** robbierut has quit IRC (Read error: Connection reset by peer) [19:30] *** DustinV has quit IRC (Ping timeout: 600 seconds) [19:31] *** robbierut has joined #archiveteam-bs [19:39] t3: "news-sites" should be "news sites", and "realised" is not a typo. [19:42] *** wabu has quit IRC (Read error: Operation timed out) [19:43] *** simon816 has quit IRC (Read error: Operation timed out) [19:43] *** dashcloud has quit IRC (Read error: Operation timed out) [19:44] JAA: Okay. [19:45] *** ivan has quit IRC (Ping timeout: 246 seconds) [19:45] *** logres133 has joined #archiveteam-bs [19:45] *** JAA has quit IRC (Ping timeout: 246 seconds) [19:45] *** c4rc4s has quit IRC (Ping timeout: 246 seconds) [19:45] *** ivan has joined #archiveteam-bs [19:45] *** svchfoo1 has quit IRC (Ping timeout: 246 seconds) [19:46] *** balrog has quit IRC (Ping timeout: 492 seconds) [19:46] *** dashcloud has joined #archiveteam-bs [19:47] *** Mayonaise has quit IRC (Read error: Operation timed out) [19:48] JAA: The FTP project appears twice: in 'Scripts only' and in 'Manual projects'. [19:54] *** Stilett0 has joined #archiveteam-bs [19:57] *** julientm has joined #archiveteam-bs [19:58] *** julientm has quit IRC (Remote host closed the connection) [19:58] *** Stilett0 has quit IRC (Ping timeout: 252 seconds) [19:58] *** Stiletto has quit IRC (Read error: Operation timed out) [19:58] *** Stiletto has joined #archiveteam-bs [19:58] *** julientm has joined #archiveteam-bs [19:59] *** Mayonaise has joined #archiveteam-bs [19:59] *** julientm has quit IRC (Remote host closed the connection) [20:01] *** julientm has joined #archiveteam-bs [20:01] The first paragraph of the "HISTORY IS OUR FUTURE" blurb on the main page abruptly switches from third- to first-person point of view. I would like to recommend some changes. [20:02] On the main wiki page (https://www.archiveteam.org/index.php?title=Main_Page)* [20:03] *** Stilett0 has joined #archiveteam-bs [20:05] *** Stiletto has quit IRC (Ping timeout: 255 seconds) [20:05] That's good priority there [20:10] *** robbierut has quit IRC (Read error: Operation timed out) [20:10] *** robbierut has joined #archiveteam-bs [20:27] *** robbierut has quit IRC (Read error: Connection reset by peer) [20:27] *** robbierut has joined #archiveteam-bs [20:29] *** Despatche has joined #archiveteam-bs [20:43] *** simon816 has joined #archiveteam-bs [20:43] *** c4rc4s has joined #archiveteam-bs [20:44] *** svchfoo1 has joined #archiveteam-bs [20:44] *** Fusl sets mode: +o svchfoo1 [20:44] *** Stiletto has joined #archiveteam-bs [20:44] *** JAA has joined #archiveteam-bs [20:44] *** Fusl sets mode: +o JAA [20:44] *** bakJAA sets mode: +o JAA [20:47] *** wabu has joined #archiveteam-bs [20:48] *** Stilett0 has quit IRC (Ping timeout: 615 seconds) [20:49] *** tuluu_ has quit IRC (Ping timeout: 265 seconds) [20:49] *** dhyan_nat has quit IRC (Read error: Operation timed out) [20:59] *** tuluu has joined #archiveteam-bs [21:07] *** balrog has joined #archiveteam-bs [21:25] *** robbierut has quit IRC (Read error: Connection reset by peer) [21:25] *** robbierut has joined #archiveteam-bs [21:27] *** kode54 has quit IRC (Quit: ZNC 1.7.2 - https://znc.in) [21:32] *** Destroyer has joined #archiveteam-bs [21:35] *** kode54 has joined #archiveteam-bs [21:43] *** d5f4a3622 has quit IRC (Read error: Connection reset by peer) [21:43] *** d5f4a3622 has joined #archiveteam-bs [22:03] *** balrog has quit IRC (Quit: Bye) [22:08] *** balrog has joined #archiveteam-bs [22:09] *** BlueMax has joined #archiveteam-bs [22:24] *** Boppen has quit IRC (Ping timeout: 186 seconds) [22:26] *** Boppen has joined #archiveteam-bs [22:26] *** Boppen has quit IRC (Read error: Connection reset by peer) [22:32] *** Exairnous has joined #archiveteam-bs [22:43] *** Exairnous has quit IRC (Ping timeout: 615 seconds) [23:25] Can someone take a look at https://www.archiveteam.org/index.php?title=Battlestar_Wiki the status went to offline [23:26] wondering what the Archiving status will be? [23:26] *** Destroyer has quit IRC (Ping timeout: 260 seconds) [23:48] julientm: Has it been archived with ArchiveBot? [23:48] no I don't think so [23:48] t3 [23:49] julientm: Oh that's a problem. Is the site permanently offline? [23:50] I believe so, but, the old admins might have a copy. They tried to relaunch the site, it was costing too much hosting. [23:50] t3 [23:52] julientm: I think the site copy can be used to somehow. We would have to communicate with the site admins. [23:52] JAA: What should be done about this? [23:53] t3 how do I verify if the archivebot arlready archived it? [23:56] julientm: See http://archive.fart.website/archivebot/viewer/?q=battlestarwiki.org [23:57] t3 thx [23:57] You can use that site to search for stuff that has been archived using ArchiveBot. [23:59] Yeah, trying to contact the owners would be a good option. [23:59] julientm: The Battlestar wiki can be saved in the form of the XML dumps or in the form of WARCs. [23:59] Offline + Not saved yet seems okay as long as there are no signs that it's lost.