#archiveteam-bs 2018-05-14,Mon

↑back Search

Time Nickname Message
00:30 🔗 Soni has quit IRC (Remote host closed the connection)
00:38 🔗 Soni has joined #archiveteam-bs
00:42 🔗 BlueMax has joined #archiveteam-bs
02:44 🔗 dashcloud I'm thinking it might be several shows?
03:03 🔗 Valentine has quit IRC (Ping timeout: 506 seconds)
03:42 🔗 qw3rty114 has joined #archiveteam-bs
03:47 🔗 qw3rty113 has quit IRC (Read error: Operation timed out)
03:49 🔗 odemg has quit IRC (Read error: Operation timed out)
03:54 🔗 Mateon1 has quit IRC (Read error: Operation timed out)
03:54 🔗 Mateon1 has joined #archiveteam-bs
03:55 🔗 ndiddy has quit IRC ()
03:59 🔗 odemg has joined #archiveteam-bs
10:57 🔗 zhongfu has joined #archiveteam-bs
11:06 🔗 BlueMax has quit IRC (Leaving)
11:53 🔗 Sk1d has joined #archiveteam-bs
14:28 🔗 MrRadar2 has quit IRC (Read error: Operation timed out)
14:30 🔗 yuitimoth has quit IRC (Ping timeout: 260 seconds)
15:02 🔗 MrRadar2 has joined #archiveteam-bs
15:03 🔗 svchfoo3 sets mode: +o MrRadar2
15:43 🔗 ola_norsk has joined #archiveteam-bs
15:44 🔗 ola_norsk isn't this "force a quick crawl" exactly what #archivebot does? :D https://archive.org/post/1090545/fee-based-enhanced-services
15:45 🔗 ola_norsk (not that it might not be a cool way to get businesses to open their wallet..)
16:11 🔗 ivan ola_norsk: IA runs Archive-It
16:15 🔗 ola_norsk oh. well there's his or her's answer then
16:25 🔗 ola_norsk https://www.irs.gov/taxonomy/term/17426
16:31 🔗 ola_norsk dangit
16:31 🔗 ola_norsk sorry, drag-n-dropped that to wrong window somehow
16:33 🔗 ola_norsk has quit IRC (leaving)
16:44 🔗 wp494 has quit IRC (Ping timeout: 260 seconds)
16:46 🔗 wp494 has joined #archiveteam-bs
16:47 🔗 svchfoo1 sets mode: +o wp494
16:48 🔗 schbirid has joined #archiveteam-bs
17:01 🔗 JAA "134 GB" of footage from the Syrian Civil War: https://old.reddit.com/r/CombatFootage/comments/8jcgb3/i_have_made_part_of_my_syrian_civil_war_archive/
17:42 🔗 godane SketchCow: any news about the tapes?
18:22 🔗 Valentine has joined #archiveteam-bs
18:33 🔗 Valentine has quit IRC (Quit: Addio, adieu, adios, aloha, arrivederci, auf Wiedersehen, au revoir, bye, bye-bye, cheerio, cheers, farewell, good)
18:37 🔗 Valentine has joined #archiveteam-bs
18:49 🔗 tyzoid has joined #archiveteam-bs
18:49 🔗 tyzoid sounds good
18:50 🔗 tyzoid (conv with JAA continuing from #archiveteam)
18:51 🔗 tyzoid So do you think pipeline might be a better option, in this case?
18:55 🔗 JAA Yeah, whenever you want to run multiple warriors in parallel, using the scripts themselves is probably a better idea in my opinion.
18:56 🔗 JAA Also if you want to support multiple projects at once, rather than only the default one.
18:57 🔗 JAA However, the downside of running scripts directly is that you have to do everything manually.
18:57 🔗 JAA Joining new projects, updating the code, etc.
18:58 🔗 JAA A possible alternative for you might be to run a VM (with Debian, Ubuntu, whichever you prefer) and then the Docker warrior multiple times inside that VM.
19:02 🔗 tyzoid that's possible too, though I feel bad about running containers in vms when I've got a container manager already xD
19:02 🔗 tyzoid so JAA: I'm thinking of setting up a pipeline instead, then it'll be able to process whatever.
19:03 🔗 JAA "Pipeline"'s a very general concept used inside the warrior but also on ArchiveBot.
19:04 🔗 tyzoid Ah, I'm referring to Archivebot pipeline
19:04 🔗 tyzoid "As of November 2017, ArchiveBot has again started accepting applications from volunteers who want to set up new pipelines"
19:05 🔗 JAA Right, but that has nothing at all to do with the warrior.
19:06 🔗 tyzoid My goal wasn't to run warrior. I've done that in the past, though. My goal is to set up something that'll be helpful, but won't require a lot of manual intervention
19:06 🔗 tyzoid multiple ip addresses is a bonus, if supported
19:07 🔗 tyzoid (I've got an entire /48 that I route portions of, and two free ipv4 addresses)
19:08 🔗 JAA ArchiveBot pipelines aren't exactly low-maintenance, unfortunately. Not as low maintenance as warrior VMs, at least.
19:09 🔗 tyzoid Hmm
19:09 🔗 JAA Also, IPv6 isn't really supported anywhere yet as far as I know. ArchiveBot definitely doesn't use it.
19:09 🔗 tyzoid I see.
19:09 🔗 tyzoid I could give root access to the container, since it'd be like a vps
19:10 🔗 tyzoid but it might just be that I'd need to run multiple warriors to take advantage of ips
19:13 🔗 tyzoid Is that a deliberate choice to not support ipv6? or is that just a thing that hasn't really been taken advantage of yet?
19:19 🔗 JAA The latter, I think.
19:20 🔗 JAA For ArchiveBot, I recently asked about what's needed for IPv6 support because I want to add it: https://github.com/ArchiveTeam/ArchiveBot/issues/315
19:23 🔗 tyzoid JAA: I assume it's because some sites might return different results on ipv4/ipv6, esp. if ipv6 is temporarily broken.
19:23 🔗 tyzoid I know it's happened on my sites. I try to maintain ipv6, but it sometimes breaks.
19:24 🔗 tyzoid shouldn't be too big an issue for larger sites, esp. if a positive result is returned
19:24 🔗 tyzoid but that's just my guess
19:30 🔗 tyzoid JAA: is there support for having multiple admins on a pipeline server? Like if something breaks, could I give key access to one of the team leads here to fix it?
19:38 🔗 JAA tyzoid: Yeah, that is an issue. However, more and more end users have IPv6 support nowadays, so arguably, that's what we should grab. Plus, that's not really an argument anyway because websites also frequently serve localised versions depending on the source IP, so the archive will at least sometimes not match what an individual user would see anyway.
19:40 🔗 JAA Sure, access to an ArchiveBot pipeline can be shared. It's really just a script running on a machine in a tmux session.
19:50 🔗 JAA tyzoid: However, we generally accept ArchiveBot pipelines only from people who have been around for a while since it's a long-term commitment. ArchiveBot pipelines need to be online continuously for months at a time.
19:52 🔗 tyzoid JAA: I've not been around this project in particular for a long time, but I've been running several services for a while
19:52 🔗 tyzoid https://status.arlm.tyzoid.com/
21:21 🔗 jschwart has quit IRC (Quit: Konversation terminated!)
21:25 🔗 t2t2 has quit IRC (Ping timeout: 260 seconds)
21:28 🔗 ivan archive.is appears to be lacking the cloudflare ddos filter or aggressive bot blocker right now
21:30 🔗 schbirid has quit IRC (Quit: Leaving)
21:32 🔗 t2t2 has joined #archiveteam-bs
21:46 🔗 tyzoid JAA: Do you run a pipeline instance yourself?
22:33 🔗 rbraun has quit IRC (Read error: Operation timed out)
23:20 🔗 me_ is now known as yipdw
23:23 🔗 rbraun has joined #archiveteam-bs
23:37 🔗 BlueMax has joined #archiveteam-bs

irclogger-viewer