#archiveteam-ot 2018-12-25,Tue

↑back Search

Time Nickname Message
00:12 🔗 BlueMax has quit IRC (Read error: Connection reset by peer)
00:25 🔗 VerfiedJ has quit IRC (Quit: Leaving)
00:35 🔗 m007a83 has joined #archiveteam-ot
00:40 🔗 Dj-Wawa has quit IRC (Quit: Connection closed for inactivity)
00:42 🔗 jrwr https://jinglepings.com/
00:42 🔗 jrwr I love the internet
00:42 🔗 jrwr This is the best thing ever
00:49 🔗 JAA "Since the led wall was being flooded with inappropriate content" lol
00:50 🔗 JAA Very nice idea!
00:50 🔗 jrwr I fucking love it
00:50 🔗 jrwr I want to set one up here locally
00:51 🔗 jrwr you don't even need that much IPv6 Space
00:51 🔗 jrwr I'm going to paint the Archive Team Warrior Logo on it
00:51 🔗 jrwr im setting it up right now
00:52 🔗 Kaz the rainbow line going across seeing to be having some packet loss
00:53 🔗 jrwr lol
00:54 🔗 jrwr damn, all my hetzner boxes have shit IPv6
00:55 🔗 jrwr ah, got one
02:31 🔗 wp494 has quit IRC (Read error: Operation timed out)
02:32 🔗 wp494 has joined #archiveteam-ot
02:32 🔗 svchfoo1 sets mode: +o wp494
02:44 🔗 Sian1468 has joined #archiveteam-ot
02:55 🔗 Sian1468 has quit IRC (Quit: Quit)
03:02 🔗 edisded I like that we've doomed ourselves before the world has even completely flipped to IPV6
03:04 🔗 kbtoo Does bing even support IPv6 yet?
03:05 🔗 edisded Maybe it's just me, but standardizing a normal end user as a /48 just seems like a silly idea
03:54 🔗 kode54 someone almost faded out the whole board for a second
04:02 🔗 kiska1 has quit IRC (Read error: Operation timed out)
04:03 🔗 kiska1 has joined #archiveteam-ot
04:51 🔗 BlueMax has joined #archiveteam-ot
04:57 🔗 odemg has quit IRC (Ping timeout: 265 seconds)
05:09 🔗 odemg has joined #archiveteam-ot
05:16 🔗 Odd0002 has joined #archiveteam-ot
06:14 🔗 wp494 has quit IRC (Ping timeout: 506 seconds)
06:15 🔗 wp494 has joined #archiveteam-ot
06:16 🔗 svchfoo1 sets mode: +o wp494
07:41 🔗 edisded Oooo
07:41 🔗 edisded Not every day this happens
07:41 🔗 edisded I got served what was supposed to be a spammy ad
07:41 🔗 edisded But the webserver appears to be pwned
08:51 🔗 BlueMax has quit IRC (Read error: Connection reset by peer)
09:50 🔗 godane has quit IRC (Ping timeout: 265 seconds)
09:57 🔗 godane has joined #archiveteam-ot
09:57 🔗 svchfoo1 sets mode: +o godane
10:55 🔗 Sian1468 has joined #archiveteam-ot
13:10 🔗 Dj-Wawa has joined #archiveteam-ot
13:31 🔗 VoynichCr is there a dump of opensubtitles or similar sites? searching in the scripts of all movies would be cool
13:35 🔗 Fusl psi
13:36 🔗 Fusl what ubuntu is that?
13:36 🔗 psi 18.04, basically fresh install
13:36 🔗 Fusl thats weird. i have a /etc/network/interfaces on mine but that could as well be because i run it with cloud-init
13:37 🔗 Fusl what you can try
13:37 🔗 Fusl `ip l l`, find your network interface name, then grep -r INTERFACENAME /etc
13:37 🔗 Fusl and try to find whatever files reference that network interface
13:38 🔗 JAA VoynichCr: We did grab a few such sites through ArchiveBot a while ago when there was a wave of shutdowns (#domtitles). But not sure if an easily searchable DB exists.
13:43 🔗 psi Fusl: I see some IPv6 related files and a file in the network folder of systemd but it seems commented out
13:43 🔗 Fusl does the server get its ip via dhcp?
13:43 🔗 Fusl (check ps fauxww | grep dhclient)
13:44 🔗 psi only gets the grep process
13:48 🔗 Fusl thats weird
13:55 🔗 Sian1468 has quit IRC (Quit: Leaving)
14:08 🔗 psi Oh
14:08 🔗 psi Fusl: https://askubuntu.com/questions/1031709/ubuntu-18-04-switch-back-to-etc-network-interfaces
14:09 🔗 Fusl oh god netplan
14:10 🔗 psi Time to install Ubuntu 16 instead :)
14:10 🔗 Fusl https://netplan.io/faq#use-pre-up-post-up-etc-hook-scripts
14:11 🔗 Fusl anyway
14:11 🔗 Fusl you dont necessarily need to put the iptables in a post-up for that interface, you can also manually run it every time after you reboot the machine
14:11 🔗 psi yikes
14:12 🔗 psi I had nothing set up yet, so I'd rather switch back to 16.04 and do it right the first time
14:12 🔗 Fusl or just use an OS that doesn't f... with its users, like debian or alpine linux :P
14:13 🔗 psi I don't have any experience with those so I'm a bit afraid to move to those
14:13 🔗 Fusl debian is literally the same as ubuntu
14:13 🔗 Fusl just without all that canonical crap installed
14:13 🔗 psi oh
14:14 🔗 psi well time to go ahead and try that then :p
14:15 🔗 JAA Grr, my auto-op script broke and I don't know why.
14:24 🔗 psi Gotta have patience for reinstalling SyS servers
15:18 🔗 wp494 has quit IRC (Ping timeout: 492 seconds)
15:19 🔗 wp494 has joined #archiveteam-ot
15:19 🔗 svchfoo1 sets mode: +o wp494
17:00 🔗 Mateon1 has quit IRC (Read error: Operation timed out)
17:02 🔗 Mateon1 has joined #archiveteam-ot
17:02 🔗 terorie has joined #archiveteam-ot
17:14 🔗 Fusl hook54321: what article?
17:17 🔗 jut Fusl: https://motherboard.vice.com/en_us/article/d3bekm/archivists-say-tumblr-ip-banned-them-for-trying-to-preserve-adult-content
17:19 🔗 Fusl ic
17:19 🔗 Fusl well there are certainly mistakes but nothing that causes bad AT reputation :D
17:21 🔗 hook54321 Fusl: Main reason I said that is they wrote what seemed to be an article designed to make IA look bad awhile ago.
17:22 🔗 Fusl did they?
17:22 🔗 Fusl i dont read news for such reasons
17:22 🔗 hook54321 looking for it, one sec
17:24 🔗 hook54321 https://motherboard.vice.com/en_us/article/nekzzq/wayback-machine-deleting-evidence-flexispy
17:25 🔗 hook54321 And then they proceeded to post the article on their Twitter feed at least five times
17:26 🔗 Fusl it was made unaccessible when they put a robots.txt on the side i guess?
17:27 🔗 hook54321 Iirc they requested that IA make it non-public, IA probably complied because the company has copyright over the material
17:29 🔗 hook54321 Also, the author of the article referenced a tweet (that they later deleted) that talks about diversifying archives, but more than half of the services mentioned in the tweet are ran by IA. https://web.archive.org/web/20180526092639/https://twitter.com/josephfcox/status/999218176364892160
17:36 🔗 hook54321 I don't know when they deleted it, but it was after it was brought up in this channel. So there's a chance they're in here.
17:38 🔗 Fusl lol that reply tweet
17:40 🔗 Dj-Wawa has quit IRC (Quit: Connection closed for inactivity)
18:02 🔗 psi Fusl: I have the IP block now, but there is no docker0 interface in /etc/network/interfaces... :/
18:02 🔗 psi it does show up in `ip l l` though
18:08 🔗 Fusl yeah, docker0 is managed by the docker daemon
18:08 🔗 Fusl you'll need to add the post-up commands into the block for your primary network interface
18:08 🔗 psi Oh I see
18:08 🔗 psi Which would probably be the eth0 IPv4 interface?
18:09 🔗 Fusl yeah
18:09 🔗 psi And do I need to restart anything after adding that line?
18:10 🔗 Fusl you can copy the post-up commands out and just run them as normal commands
18:10 🔗 Fusl that will do it
18:10 🔗 psi Starting with `iptables` I assume
18:11 🔗 Fusl yeah
18:11 🔗 Fusl once you've configured iptables to balance across your ips or prefixes, you can use `docker run --rm -ti alpine sh -c 'for i in $(seq 20); do wget -qO- https://ipinfo.io/ip; done'` to verify that your iptables rules are working properly
18:12 🔗 Fusl it should give you the public ips back the docker container got after each request
18:13 🔗 Fusl_ has joined #archiveteam-ot
18:13 🔗 psi Hm, it's getting the same IP every time, but it is one from my new IP block
18:13 🔗 Fusl how do your iptables rules look like?
18:15 🔗 psi IPs censored, obviously https://www.irccloud.com/pastebin/N8CInW3c/
18:15 🔗 psi The only other ones in /etc/network/interfaces are the loopback interface and the IPv6 eth0
18:17 🔗 psi Oh, the test command is now alternating between 5.6.7.3 and 5.6.7.10
18:17 🔗 Fusl oh yeah thats what it should do
18:17 🔗 Fusl if it does that, everythings working
18:18 🔗 psi Shouldn't all 16 IPs technically be used?
18:19 🔗 Fusl technically yes, but "randomness" in iptables is kinda weird. once you get some threads up and running, they will alternate between all ips
18:19 🔗 psi alrighty
18:21 🔗 Fusl once you have some threads running you can check with `tcpdump -plnn -i eth0 'tcp[tcpflags] == tcp-syn' | awk -Winteractive '{print $3}' | awk -Winteractive -F. '{print $1"."$2"."$3"."$4}'` to see source ip addresses for tcp connects
18:22 🔗 psi I see some IP addresses come by that I don't even recognise but most of them seem to be from my IP block
18:22 🔗 terorie has quit IRC (Remote host closed the connection)
18:22 🔗 Fusl those are inbound connections then
18:23 🔗 psi inb4 russia's trying to hack me
18:23 🔗 Fusl try this instead `tcpdump -Qout -plnn -i eth0 'tcp[tcpflags] == tcp-syn' | awk -Winteractive '{print $3}' | awk -Winteractive -F. '{print $1"."$2"."$3"."$4}'`
18:24 🔗 psi `Unable to write output: broken pipe`
18:24 🔗 psi oh my bad
18:24 🔗 psi Yep that seems plenty random
18:25 🔗 Fusl if you want fuller randomness you can still go ahead and write a custom iptables target chain with -m statistic --mode random or --mode nth
18:25 🔗 terorie has joined #archiveteam-ot
18:26 🔗 psi I hardly know what I'm doing anyway so I'll just leave it alone x)
18:26 🔗 Fusl kk
18:27 🔗 psi But now they have to block a /28 block instead of just one IP >:)
18:27 🔗 psi Also, regarding the multiple IP blocks idea, it used to be the case that you have multiple --to-source params and it would round-robin between them
18:28 🔗 psi But they removed that for whatever reason
18:31 🔗 psi Oh, it was actually removed in the Linux kernel
18:48 🔗 psi Hey Fusl does the `touch STOP` thing work when `docker exec -it`'d in containers running your file
18:49 🔗 Fusl it should
18:49 🔗 psi Hm, guess the tasks just take a while then
18:50 🔗 Fusl you can check with `docker logs -f <id>`
18:51 🔗 psi that's a lot of logs
18:52 🔗 psi But yeah it doesn't seem like it's getting new jobs
18:57 🔗 terorie has quit IRC (Remote host closed the connection)
18:59 🔗 Fusl not-quick and dirty script for even distribution of source nat ips http://xor.meo.ws/nbx3sPbsWx2b0TJWVRPNgnx0UiYb7DHj.txt
19:00 🔗 Fusl requires all your "possible" outbound ip addresses to be added to the primary network interface
19:07 🔗 psi meh
19:16 🔗 ubahn has joined #archiveteam-ot
19:22 🔗 psi Hm
19:23 🔗 psi It's probably easier to do this the Docker Swarm way
19:24 🔗 ubahn has quit IRC (Quit: ubahn)
19:30 🔗 Odd0002 has quit IRC (ZNC - http://znc.in)
19:33 🔗 VerfiedJ has joined #archiveteam-ot
19:33 🔗 schbirid oh shit apparently i can get gigabit internet at home
19:33 🔗 schbirid shiiiit
19:34 🔗 schbirid 50mbit/s upstream lol
19:36 🔗 Odd0002 has joined #archiveteam-ot
19:37 🔗 Odd0002 has quit IRC (Client Quit)
19:41 🔗 Odd0002 has joined #archiveteam-ot
19:53 🔗 ubahn has joined #archiveteam-ot
19:58 🔗 ubahn has quit IRC (Client Quit)
20:57 🔗 terorie has joined #archiveteam-ot
21:02 🔗 terorie has quit IRC (Ping timeout: 268 seconds)
22:04 🔗 terorie has joined #archiveteam-ot
22:05 🔗 s4t has joined #archiveteam-ot
22:09 🔗 terorie has quit IRC (Ping timeout: 268 seconds)
22:16 🔗 s4t has quit IRC (Quit: s4t)
22:16 🔗 terorie has joined #archiveteam-ot
22:33 🔗 BlueMax has joined #archiveteam-ot
22:46 🔗 Dj-Wawa has joined #archiveteam-ot
23:18 🔗 Odd0002 has quit IRC (ZNC - http://znc.in)
23:21 🔗 Odd0002 has joined #archiveteam-ot
23:23 🔗 silas has joined #archiveteam-ot
23:35 🔗 silas I just finished downloading an FTP site as a WARC using wget. It made a WARC file and a normal folder with the stuff it downloaded, and my question is if I zip up the folder, will the Wayback Machine still be able to ingest the WARC properly? I feel like it's a kinda dumb question but I want to be sure since it's all a few hundred GB altogether and I don't want to mess anything up
23:38 🔗 silas And just so it's clear I don't mean zipping up the warc with it, I just mean zipping up the subfolder wget makes that has all the files it's downloaded in it.

irclogger-viewer