#archiveteam-bs 2017-06-06,Tue

↑back Search

Time Nickname Message
00:01 🔗 gui7 has joined #archiveteam-bs
00:02 🔗 gui7 has quit IRC (Client Quit)
00:03 🔗 BlueMaxim has joined #archiveteam-bs
00:21 🔗 JRWR-Thin has joined #archiveteam-bs
00:27 🔗 BlueMaxim has quit IRC (Quit: Leaving)
00:31 🔗 Whopper_ has quit IRC (Ping timeout: 246 seconds)
00:31 🔗 Whopper has joined #archiveteam-bs
00:35 🔗 JRWR-Thin so far I'm impressed with the speed of the scaleway ARM64-2GB for a warrior
00:35 🔗 JRWR-Thin just wish it has more disk space
00:35 🔗 JRWR-Thin does run-pipeline handle low diskspace?
00:37 🔗 pizzaiolo has quit IRC (Remote host closed the connection)
01:07 🔗 MrRadar Yo, if you're running SPUF via scripts please make sure to update them. It's a wall of Warriors over on the tracker right now...
01:07 🔗 MrRadar (And no, the project is not done yet.)
01:10 🔗 ndiddy has quit IRC (Read error: Operation timed out)
01:20 🔗 j08nY has quit IRC (Quit: Leaving)
01:22 🔗 JRWR-Thin has quit IRC (Ping timeout: 268 seconds)
01:28 🔗 JRWR-iPAD has joined #archiveteam-bs
01:35 🔗 username1 has joined #archiveteam-bs
01:35 🔗 JRWR-iPAD has quit IRC (Ping timeout: 268 seconds)
01:38 🔗 schbirid2 has quit IRC (Read error: Operation timed out)
01:40 🔗 JRWR-iPAD has joined #archiveteam-bs
02:05 🔗 ZexaronS has quit IRC (Leaving)
02:17 🔗 joepie91 JRWR: you can add additional disk space, no?
02:18 🔗 JRWR its a OVH Box
02:18 🔗 JRWR So I could spin up another box and move the DNS
02:30 🔗 JRWR But I can't afford it
03:11 🔗 joepie91 JRWR: huh? you said Scaleway
03:11 🔗 joepie91 not OVH :P
03:11 🔗 joepie91 and Scaleway has block storage stuff iirc
03:11 🔗 JRWR Wait
03:11 🔗 JRWR What is the topic, I was talking about the Rsync Ingress I'm running
03:13 🔗 joepie91 [02:35] <JRWR-Thin> so far I'm impressed with the speed of the scaleway ARM64-2GB for a warrior
03:13 🔗 joepie91 [02:35] <JRWR-Thin> just wish it has more disk space
03:33 🔗 yuitimoth has quit IRC (Remote host closed the connection)
03:33 🔗 yuitimoth has joined #archiveteam-bs
03:49 🔗 r3c0d3x has joined #archiveteam-bs
04:15 🔗 MrRadar arkiver xmc: Someone from pixiv is asking us to throttle our requests a bit
04:16 🔗 MrRadar Since our grab is triggering their automatic anti-DDoS systems
04:17 🔗 Odd0002 I think this is day 3 of my yahoo answers thread running
04:17 🔗 MrRadar Kaz, GLaDOS: See above ^^^
04:18 🔗 MrRadar Over in #savepixiv
04:44 🔗 BlueMaxim has joined #archiveteam-bs
04:59 🔗 Sk1d has quit IRC (Ping timeout: 250 seconds)
04:59 🔗 dashcloud has quit IRC (Ping timeout: 492 seconds)
05:06 🔗 Sk1d has joined #archiveteam-bs
05:31 🔗 JRWR-iPAD Hrm
05:31 🔗 JRWR-iPAD We could use a tracker admin
05:31 🔗 JRWR-iPAD need to put the brakes on pixiv
05:31 🔗 JRWR-iPAD 40RPS has beerequested
05:32 🔗 Odd0002 ok, I'll pause mine for now
05:33 🔗 JRWR-iPAD Ill stop mine as well
06:00 🔗 SHODAN_UI has joined #archiveteam-bs
06:36 🔗 bsmith093 has quit IRC (Read error: Operation timed out)
07:23 🔗 godane so i'm close to going past last month number of items
07:24 🔗 godane i have like another 100 items to upload to past it
07:32 🔗 SHODAN_UI has quit IRC (Remote host closed the connection)
07:34 🔗 tuluu has quit IRC (Ping timeout: 260 seconds)
07:52 🔗 bsmith093 has joined #archiveteam-bs
08:07 🔗 bsmith093 has quit IRC (Ping timeout: 245 seconds)
08:15 🔗 BlueMaxim has quit IRC (Read error: Operation timed out)
08:16 🔗 BlueMaxim has joined #archiveteam-bs
08:25 🔗 bsmith093 has joined #archiveteam-bs
08:28 🔗 godane http://www.npr.org/sections/alltechconsidered/2017/06/03/529155865/videotapes-are-becoming-unwatchable-as-archivists-work-to-save-them
08:39 🔗 JRWR-iPAD has quit IRC (Quit: Page closed)
08:45 🔗 jiphex has joined #archiveteam-bs
09:40 🔗 BlueMaxim has quit IRC (Quit: Leaving)
09:50 🔗 j08nY has joined #archiveteam-bs
10:02 🔗 ZexaronS has joined #archiveteam-bs
10:45 🔗 Stiletto has quit IRC (Read error: Operation timed out)
12:14 🔗 Aoede I did a scrape of literotica.com, let me know if there's any issues with it.
12:14 🔗 Aoede https://archive.org/details/literotica.com_2017-04
14:16 🔗 pizzaiolo has joined #archiveteam-bs
14:21 🔗 JAA This may be of interest to some of you: https://www.reddit.com/r/DataHoarder/comments/6fku0s/2873_nes_roms_the_eye/
14:33 🔗 alfie has quit IRC (Ping timeout: 260 seconds)
14:44 🔗 alfie has joined #archiveteam-bs
15:17 🔗 SHODAN_UI has joined #archiveteam-bs
15:17 🔗 pizzaiolo has quit IRC (Read error: Connection reset by peer)
15:19 🔗 pizzaiolo has joined #archiveteam-bs
16:08 🔗 username1 is now known as spirit
16:08 🔗 spirit JAA: probably better to use the standard sets
16:09 🔗 spirit Aoede: thanks >:)
16:20 🔗 Aoede spirit: not like I have anything better to do with my time :P
16:20 🔗 spirit well, in about 27 minutes I do
16:23 🔗 Aoede ?
16:28 🔗 spirit says wget
16:29 🔗 Aoede oh :D
16:32 🔗 Aoede It's around 11GB uncompressed
16:42 🔗 pizzaiolo has quit IRC (Quit: pizzaiolo)
16:42 🔗 joepie91 I was just pointed at this very, very interesting project: http://formats.kaitai.io/ -- cc SketchCow
16:43 🔗 joepie91 essentially standardized format specifications for a crapload of (binary) formats
16:43 🔗 joepie91 with parser and chart generation and whatnot
16:43 🔗 joepie91 (though the crapload is not yet big enough :P)
16:45 🔗 MrRadar Someone with an account on the File Formats wiki should add a link there: http://fileformats.archiveteam.org/wiki/Main_Page
16:47 🔗 JAA Interesting. Shall we throw it into ArchiveBot just in case?
17:46 🔗 bmcginty has quit IRC (Ping timeout: 268 seconds)
17:47 🔗 icedice has joined #archiveteam-bs
17:52 🔗 bmcginty has joined #archiveteam-bs
17:59 🔗 Pudsey has joined #archiveteam-bs
18:04 🔗 Pudsey has quit IRC (Remote host closed the connection)
18:36 🔗 JRWR-iPAD has joined #archiveteam-bs
18:44 🔗 SHODAN_UI has quit IRC (Remote host closed the connection)
19:43 🔗 Honno has quit IRC (Quit: Leaving)
19:48 🔗 Aranje has joined #archiveteam-bs
20:02 🔗 kristian_ has joined #archiveteam-bs
20:03 🔗 JAA I'm looking into archiving Steam Greenlight. Some of it looks pretty straightforward, other things will be really annoying.
20:06 🔗 Odd0002 what sort of stuff is there to archive? Comments and pages?
20:09 🔗 JAA The list of games which were released through Greenlight, announcements plus comments to those, and two discussion forums.
20:09 🔗 JAA As far as I can see
20:11 🔗 Odd0002 ok
20:11 🔗 JAA The announcement comments are the most annoying thing I've seen so far as they're based entirely on JavaScripted POST requests.
20:19 🔗 MrRadar Bleh
20:36 🔗 ZexaronS has quit IRC (Read error: Connection reset by peer)
20:46 🔗 ZexaronS has joined #archiveteam-bs
20:48 🔗 SHODAN_UI has joined #archiveteam-bs
21:58 🔗 spirit has quit IRC (Quit: Leaving)
21:59 🔗 dashcloud has joined #archiveteam-bs
22:07 🔗 JAA Ok, so those POST requests aren't actually necessary strictly speaking since all information can also be accessed directly as a static HTML page. That doesn't mean I won't try archiving that "API" as well, of course.
22:08 🔗 JAA Each game released through Greenlight obviously has its own page, including a discussion forum etc. I think I'll skip those for now.
22:09 🔗 arkiver do you have a list of games
22:09 🔗 arkiver or IDs
22:09 🔗 JAA Not yet, but I guess I can make one, why?
22:09 🔗 arkiver well for archiving
22:10 🔗 JAA Well yeah, but I don't see those games going anywhere anytime soon. It's just the whole Greenlight framework which will disappear, as I understand it.
22:10 🔗 JAA It would certainly be a good idea to grab all of Steam at some point though.
22:10 🔗 arkiver ah
22:10 🔗 arkiver at some point yeah
22:11 🔗 arkiver any idea how big this is?
22:11 🔗 arkiver greenlight I mean
22:11 🔗 JRWR 14k games overall over the course of its lifetime
22:11 🔗 JRWR thats including all the crap ones
22:12 🔗 arkiver ah
22:12 🔗 arkiver good enough for archivebot I think
22:12 🔗 JAA Not big. 16k games and 6k forum threads are the main part.
22:12 🔗 arkiver right yeah, so a few GB only
22:13 🔗 JAA I don't think ArchiveBot will handle this very well. It's spread across various directories on the steamcommunity.com domain.
22:13 🔗 JAA A specific wpull with the relevant --accept-regex rules will work better, I think.
22:14 🔗 arkiver yep
22:14 🔗 JRWR-iPAD like a mini archivebocx
22:14 🔗 JRWR-iPAD bot*
22:14 🔗 arkiver :P
22:14 🔗 JAA :-)
22:14 🔗 JAA JustAnotherArchivebot
22:18 🔗 JAA Hmm, actually, it may be necessary to grab the games as well. Still not big though.
22:20 🔗 JAA Just painful
22:34 🔗 SHODAN_UI has quit IRC (Remote host closed the connection)
22:41 🔗 ZexaronS- has joined #archiveteam-bs
22:43 🔗 ZexaronS has quit IRC (Read error: Operation timed out)
22:54 🔗 BartoCH has quit IRC (Ping timeout: 260 seconds)
23:13 🔗 kristian_ has quit IRC (Quit: Leaving)
23:14 🔗 yipdw holy shit http://codeology.braintreepayments.com/archiveteam/archivebot
23:14 🔗 yipdw this is a sweet repo visualization
23:15 🔗 JRWR IS THAT FLYING SPAGHETTI MONSTER?
23:16 🔗 yipdw what I really like is that the objects used to represent the repo aren't random
23:16 🔗 yipdw Python code, for example, always shows up as those purple pyramidial-ish things
23:16 🔗 yipdw stuff recognized as Makefile directives show up as those spindly structures
23:16 🔗 JRWR Archive bot is FSM
23:16 🔗 JRWR Legit thats fucking GSM
23:16 🔗 JRWR FSM*
23:17 🔗 yipdw so you develop a visual (and, if this were to be e.g. 3D-printed, tactile/material) vocabulary for a repo's composition
23:18 🔗 yipdw here's seesaw: http://codeology.braintreepayments.com/archiveteam/seesaw-kit
23:19 🔗 JRWR my god, its ALIVE
23:19 🔗 JRWR http://codeology.braintreepayments.com/archiveteam/glowing-computing-machine
23:20 🔗 yipdw anything with a lot of spindly things sticking out of it immediately signals "the build system comprises a substantial portion of the code"
23:20 🔗 yipdw that's really cool
23:22 🔗 JRWR has quit IRC (Quit: Page closed)
23:23 🔗 JRWR has joined #archiveteam-bs
23:24 🔗 icedice has quit IRC (Quit: Leaving)
23:25 🔗 JRWR oh
23:25 🔗 JRWR oh my god
23:25 🔗 JRWR http://codeology.braintreepayments.com/featured/torvalds/linux
23:25 🔗 JRWR Linux is AMAZING
23:29 🔗 voidsta that's pretty cool
23:30 🔗 JRWR http://codeology.braintreepayments.com/featured/microsoft/typescript
23:30 🔗 JRWR looks like a bat
23:31 🔗 voidsta haha, yep
23:49 🔗 BlueMaxim has joined #archiveteam-bs
23:51 🔗 zenguy has quit IRC (Ping timeout: 370 seconds)

irclogger-viewer