#urlteam 2015-12-16,Wed

↑back Search

Time Nickname Message
00:01 πŸ”— svchfoo1 has joined #urlteam
00:02 πŸ”— svchfoo3 sets mode: +o svchfoo1
00:03 πŸ”— dashcloud has quit IRC (Read error: Operation timed out)
00:04 πŸ”— dashcloud has joined #urlteam
00:05 πŸ”— svchfoo1 sets mode: +o dashcloud
00:30 πŸ”— dashcloud has quit IRC (Read error: Connection reset by peer)
00:31 πŸ”— dashcloud has joined #urlteam
00:31 πŸ”— svchfoo1 sets mode: +o dashcloud
01:00 πŸ”— Silvan has quit IRC (Read error: Operation timed out)
01:02 πŸ”— dashcloud has quit IRC (Read error: Operation timed out)
01:06 πŸ”— dashcloud has joined #urlteam
01:06 πŸ”— svchfoo1 sets mode: +o dashcloud
01:07 πŸ”— bwn_ has quit IRC (Read error: Operation timed out)
01:10 πŸ”— bwn has joined #urlteam
01:28 πŸ”— JesseW has joined #urlteam
01:28 πŸ”— svchfoo1 sets mode: +o JesseW
01:33 πŸ”— JesseW up'ed the queue for da-gd to 10 items
01:34 πŸ”— JesseW 7 results found so far, in ~ 120,000 checks
01:34 πŸ”— JesseW hm, if they were evenly distributed, we should have found more like 12 by now
01:34 πŸ”— JesseW apparently they aren't
01:39 πŸ”— JesseW I think we have gotten everything from dld-bz
01:39 πŸ”— JesseW I've stopped it for now
01:52 πŸ”— dashcloud has quit IRC (Read error: Operation timed out)
01:54 πŸ”— dashcloud has joined #urlteam
01:55 πŸ”— svchfoo1 sets mode: +o dashcloud
02:22 πŸ”— JesseW started ccl.hu -- we'll see if it picks up *anything* useful
02:44 πŸ”— JesseW starting chilp-it -- 6 (and 7) character non-incremental lowercase alpha & digits. So a search space of about 2 billion for the 6 character ones, and 78 billion for the 7 character ones.
02:45 πŸ”— JesseW we can certainly do the 6 character ones; maybe not the 7 characters
02:47 πŸ”— aaaaaaaaa go big or go home
02:48 πŸ”— JesseW I mean, we can certainly *start* the 7 character ones -- but considering in about a year's running we've only done about 21 billion checks total...
02:50 πŸ”— aaaaaaaaa Well, I was referring to the whole idea that you can't finish things you don't start. But in all seriousness, the tracker might have to be a bit more robust to handle the traffic necessary for a grab that big.
02:50 πŸ”— JesseW and we'd need to boost the number of people running warriors by quite a bit
02:54 πŸ”— W1nterFox has quit IRC (Ping timeout: 506 seconds)
03:00 πŸ”— JesseW sadly, it looks like ccl.hu died before we could work on it. :-(
03:01 πŸ”— JesseW 100,000 checks and nothing useful found.
03:08 πŸ”— W1nterFox has joined #urlteam
03:35 πŸ”— bwn has quit IRC (Read error: Operation timed out)
03:37 πŸ”— winterfox has joined #urlteam
03:40 πŸ”— W1nterFox has quit IRC (Read error: Operation timed out)
04:12 πŸ”— JesseW So there are 6 url shortening services that signed up with 301works, but have since died. We need to ping IA (and the other folks involved with 301works) and get the material un-darked.
04:21 πŸ”— dashcloud has quit IRC (Read error: Operation timed out)
04:24 πŸ”— dashcloud has joined #urlteam
04:25 πŸ”— svchfoo1 sets mode: +o dashcloud
04:51 πŸ”— aaaaaaaaa has quit IRC (Leaving)
05:23 πŸ”— Mayonaise has quit IRC (Ping timeout: 606 seconds)
05:23 πŸ”— zhongfu_ has joined #urlteam
05:24 πŸ”— dashcloud has quit IRC (Ping timeout: 606 seconds)
05:24 πŸ”— dashcloud has joined #urlteam
05:24 πŸ”— svchfoo1 sets mode: +o dashcloud
05:26 πŸ”— Mayonaise has joined #urlteam
05:27 πŸ”— zhongfu has quit IRC (Ping timeout: 606 seconds)
05:33 πŸ”— JesseW chfoo -- it'd be nice to get this PR deployed sooner than later, as it would stop migre-me from cluttering up the error_reports, and having to skip a bunch of items (which we'll have to go back through and catch later)
05:34 πŸ”— JesseW and apparently something broke the error reports display :-(
05:35 πŸ”— chfoo _tt_tmp = reverse_url('admin.error_reports')+'?project_id=' + report['project'] # admin/overview/error_reports.html:40 (via admin/base.html:25, base.html:13)
05:35 πŸ”— chfoo TypeError: Can't convert 'NoneType' object to str implicitly
05:36 πŸ”— JesseW lol. :-(
05:36 πŸ”— JesseW PR coming asap
05:36 πŸ”— chfoo does the anti regex need more code? does doing (http://example.com/)|(^$) make sense?
05:37 πŸ”— chfoo or perhaps ^\s*$
05:38 πŸ”— JesseW I don't understand what you mean...? Yes, a good way to enable the tolerate_missing_Location feature would be ^$ if that's the only pattern you want to match, or something_else.php|^$ if you wanted to match both...
05:38 πŸ”— JesseW but I don't think it needs "more code"...?
05:39 πŸ”— chfoo i mean why does it need a pull request for more logic? i can't see why the existing code won't work with adding ^$ to the option
05:42 πŸ”— JesseW ah, because it doesn't get into the existing branch if there is no Location header at all
05:44 πŸ”— chfoo oh, ok. i see it now
05:45 πŸ”— JesseW it works if the Location header is present, but empty
05:45 πŸ”— JesseW it (already) works, I mean
05:45 πŸ”— chfoo right
05:46 πŸ”— chfoo ok, seems good. you can merge and let me know when you have the error page thing fixed
05:46 πŸ”— chfoo and i'll update the code on the tracker
05:47 πŸ”— JesseW ok
05:52 πŸ”— ahrain has quit IRC (Remote host closed the connection)
05:59 πŸ”— JesseW chfoo: as GitHub83 mentioned, I've pushed a fix now.
06:00 πŸ”— chfoo ok, updating the tracker
06:01 πŸ”— JesseW excellent, thanks
06:02 πŸ”— xmc has joined #urlteam
06:02 πŸ”— swebb sets mode: +o xmc
06:04 πŸ”— JesseW let me know when the tracker comes back up
06:08 πŸ”— chfoo JesseW: everything should be ok now
06:08 πŸ”— GitHub97 has joined #urlteam
06:08 πŸ”— GitHub97 has left
06:08 πŸ”— dashcloud has quit IRC (Read error: Operation timed out)
06:09 πŸ”— JesseW cool, looks good
06:12 πŸ”— dashcloud has joined #urlteam
06:12 πŸ”— svchfoo3 sets mode: +o dashcloud
06:35 πŸ”— JesseW Atluxity: getting close to the top 10! Only 31 million scans to go. :-)
06:46 πŸ”— Atluxity woho :D
06:51 πŸ”— JesseW of course, you still have more than 3 billion scans to reach the top spot... ;-/
06:51 πŸ”— cechk01 has quit IRC (Read error: Connection reset by peer)
06:59 πŸ”— Atluxity I think I'm gaining in... :P
06:59 πŸ”— JesseW yeah, he doesn't seem to be active lately
07:08 πŸ”— cybersec How long has this project been running for?
07:08 πŸ”— cybersec were those 3.7 billion scans over the course of months/years, or has it only been going for a little bit?
07:08 πŸ”— cybersec also has he ever popped in IRC before?
07:09 πŸ”— cybersec scratch that, just realized he's in the channel right now haha
07:09 πŸ”— JesseW Ah, terroroftinytown, the current tracker, is only a year old.
07:10 πŸ”— JesseW Specifically, the earliest daily dumps from it were on Nov 6, 2014.
07:10 πŸ”— JesseW Before it, there were other tools used to produce the old dump, but they aren't reflected in the tracker totals, AFAIK.
07:11 πŸ”— cybersec ah, I see
07:11 πŸ”— cybersec is there anything known about how johtso scans so crazily fast? haha
07:12 πŸ”— JesseW cybersec: the rumor I heard is he was running it on the side of bitcoin mining equipment
07:12 πŸ”— cybersec oh yeah! sorry, I forgot you told me that the other day
07:12 πŸ”— * JesseW is going to sleep now. G'night, all -- if the tracker blows up, please put out the fire.
07:13 πŸ”— cybersec that's pretty awesome
07:13 πŸ”— cybersec night!
07:23 πŸ”— JesseW has quit IRC (Leaving.)
08:46 πŸ”— bwn has joined #urlteam
10:00 πŸ”— asdf has quit IRC (Ping timeout: 252 seconds)
10:51 πŸ”— winterfox I dont see why johtso's setup would need bitcoin mining hardware. From what I can tell you just need lots of IP addresses so shorteners dont block you for spamming them
11:02 πŸ”— Atluxity a mining operation implies to me a small server-park
11:02 πŸ”— Atluxity perhaps with public ip's
11:44 πŸ”— cybersec has quit IRC (Ping timeout: 483 seconds)
13:10 πŸ”— dashcloud has quit IRC (Read error: Operation timed out)
13:12 πŸ”— dashcloud has joined #urlteam
13:13 πŸ”— svchfoo1 sets mode: +o dashcloud
14:27 πŸ”— winterfox has quit IRC (Remote host closed the connection)
15:15 πŸ”— cybersec has joined #urlteam
15:24 πŸ”— dashcloud has quit IRC (Read error: Operation timed out)
15:27 πŸ”— dashcloud has joined #urlteam
15:27 πŸ”— svchfoo1 sets mode: +o dashcloud
15:33 πŸ”— cybersec has quit IRC (Quit: Sorry, fuckers.)
15:36 πŸ”— dashcloud has quit IRC (Ping timeout: 252 seconds)
15:36 πŸ”— dashcloud has joined #urlteam
15:36 πŸ”— svchfoo1 sets mode: +o dashcloud
15:49 πŸ”— phuzion JW_work: I think I might have figured out why we're seeing a less-than-even distribution of urls on da.gd
15:50 πŸ”— cechk01 has joined #urlteam
15:52 πŸ”— phuzion Basically, there's some number of shorturls, greater than 30,000, that are "disabled" meaning they return 404 despite being "valid" URLs. Basically, da.gd was the victim of a spambot attack at some point, and my buddy disabled the URLs that were leading to viagra spam sites.
15:52 πŸ”— Start has quit IRC (Quit: Disconnected.)
17:05 πŸ”— jornane Hi! I just got a report from my organisations CERT. They have logged that my Warrior instance (which runs there with permission, naturally) is performing DNS lookups that are associated with malware.
17:06 πŸ”— Atluxity did they specify?
17:07 πŸ”— jornane The mail said "..bigfoot1942.sektori.org….." without quotes, it is part of a suricata log.
17:07 πŸ”— Atluxity strange
17:08 πŸ”— jornane Does the warrior check URLs it finds?
17:12 πŸ”— bwn has quit IRC (Read error: Connection reset by peer)
17:12 πŸ”— bwn has joined #urlteam
17:13 πŸ”— Start has joined #urlteam
17:16 πŸ”— phuzion jornane: I don't believe that the warrior does anything along those lines, but let me pcap a warrior running urlteam for a minute and see
17:29 πŸ”— JesseW has joined #urlteam
17:29 πŸ”— svchfoo1 sets mode: +o JesseW
17:39 πŸ”— phuzion jornane: It doesn't appear that the urlteam scripts do anything like a dns lookup or http request to the actual site itself, just to the url shortener. I could be wrong, though.
17:40 πŸ”— JesseW they do not touch the actual site, that's correct.
17:41 πŸ”— jornane phuzion: I just started DNS logging myself with a Suricata of my own, so hopefully I'll find out where this comes from. Since the warrior was my latest addition to the network, I suspected it first. Looks like (hope ;-)) I was wrong.
17:42 πŸ”— phuzion jornane: Your CERT team said that the malicious DNS lookup was from that warrior's IP address? Does that VM share an IP with anything/anyone else?
17:42 πŸ”— JesseW but DNS queries are not the same as contacting the site. Has your warrior been running any other project besides urlteam?
17:44 πŸ”— JesseW has quit IRC (Leaving.)
17:46 πŸ”— jornane phuzion: yes, but it's even harder to explain the requests from the other hosts sharing the IP
17:46 πŸ”— jornane Most of my projects run IPv6-only, only very special projects that require IPv4 do get access to my NAT
17:46 πŸ”— phuzion Do any of the hosts sharing the IP run windows?
17:46 πŸ”— jornane No
17:48 πŸ”— phuzion I'd pcap the vm host (so you can figure out which VM it came from) and ask your CERT team to give you time and date stamps of any further hits so you can narrow it down.
17:51 πŸ”— jornane phuzion: doing that now, so far the results look OK, just that during the last quarter the warrior has looked up youtube.com and twitter.com, those aren't listed on http://archiveteam.org/index.php?title=Urlteam
17:51 πŸ”— phuzion https://github.com/ArchiveTeam/terroroftinytown-client-grab/blob/master/pipeline.py#L42-L47
17:52 πŸ”— jornane check, then it makes sense :) I saw the cheeseburger as well but assumed it had a redirect service :P
18:04 πŸ”— JW_work1 has joined #urlteam
18:04 πŸ”— JW_work has quit IRC (Read error: Operation timed out)
18:19 πŸ”— dashcloud has quit IRC (Read error: Operation timed out)
18:20 πŸ”— dashcloud has joined #urlteam
18:20 πŸ”— svchfoo1 sets mode: +o dashcloud
18:25 πŸ”— cechk01 has quit IRC (Read error: Connection reset by peer)
18:36 πŸ”— Start has quit IRC (Quit: Disconnected.)
19:02 πŸ”— bwn has quit IRC (Read error: Operation timed out)
19:11 πŸ”— JW_work1 has quit IRC (Leaving.)
19:12 πŸ”— JW_work has joined #urlteam
19:22 πŸ”— Start has joined #urlteam
19:23 πŸ”— bwn has joined #urlteam
19:38 πŸ”— aaaaaaaaa has joined #urlteam
19:38 πŸ”— swebb sets mode: +o aaaaaaaaa
20:06 πŸ”— Start has quit IRC (Quit: Disconnected.)
20:48 πŸ”— Start has joined #urlteam
20:48 πŸ”— Start has quit IRC (Client Quit)
21:04 πŸ”— Start has joined #urlteam
21:10 πŸ”— asdf has joined #urlteam
21:41 πŸ”— aaaaaaaa_ has joined #urlteam
21:41 πŸ”— aaaaaaaaa has quit IRC (Read error: Connection reset by peer)
21:41 πŸ”— swebb sets mode: +o aaaaaaaa_
21:44 πŸ”— aaaaaaaa_ is now known as aaaaaaaaa
22:16 πŸ”— Start has quit IRC (Quit: Disconnected.)
23:24 πŸ”— asdf this url shortener looks mysterious https://soli.dm/
23:24 πŸ”— asdf lol
23:38 πŸ”— JW_work nice big letters
23:38 πŸ”— JW_work feel free to add it anyway
23:39 πŸ”— Start has joined #urlteam
23:44 πŸ”— WinterFox has joined #urlteam

irclogger-viewer