[00:08] *** Zerote has quit IRC (Ping timeout: 260 seconds) [00:17] 35.200.144.89 leaving that there for later investigations [00:25] *** Zerote has joined #urlteam [00:33] Hello Zerote what brings you here? [00:35] *** Smiley has joined #urlteam [00:35] Hiya Smiley [00:36] *** SmileyG has quit IRC (Read error: Operation timed out) [00:37] *** tech234a has joined #urlteam [00:39] Just lurking, mostly [00:39] Fair enough [00:39] I am going to try and welcome people when I can [00:40] welcome tech234a [01:56] *** Zerote has quit IRC (Ping timeout: 260 seconds) [02:05] Thanks [02:05] So what brings you here friend? [02:06] I was around for G+, and goo.gl was shutting down and I wanted to see what was going on [02:07] Do you want an explanation or rundown roughly of the project or did the wiki do a good enough job? [02:08] From what I understand, this project is working on storing copies of short URL redirection information on IA [02:08] Thats basically it [02:08] I also believe that goo.gl URLs are not being archived yet (?) [02:08] Right now it is bruteforced and stored in txt documents [02:08] Ahh... got it [02:09] Actually we should be starting on the goo.gl project soon [02:09] Cool [02:09] The documents are released properly if the shortener goes down but are still available in the mean time as the torrents [02:10] Ok [02:10] We plan to one day have these as Dark Warc files that will be ingested into wayback if a shortener goes down [02:10] Good to know [02:10] (but I dont have the coding knowledge or time to do it) Right now I am supposed to be doing an assessment [02:11] Of goo.gl? [02:11] Though if you find any new url shorteners that arent on the wiki feel free to drop them here and ping me or one of the other ops [02:11] No a school assignment lol [02:11] Ah... cool [02:12] Do you need help discovering URLs too? [02:12] I mean if you have new url shorteners then just drop them here. [02:12] Check the wiki and the unsorted page of the wiki [02:12] Ok [02:12] Urlteam and Urlteam/Unsorted [02:12] I think they are called [02:12] Thanks [02:13] all good [02:13] I am still trying to figure out where Cyrillic and Emojis fit into alphabetical order as well so if you have any insights lol [02:14] I'm not sure about that myself :) [02:14] actually tech234a if you are bored with time on your hands you can find out which Cyrillic Alphabets http://чоч.рф/ [02:14] http://сёр.рф/ use [02:14] Those 2 url shorteners [02:14] I have been lazy about doing it myself so its up to you [02:15] I'm a little busy right now, but I have a script that discovers URLs from CommonCrawl - I can grab those for you [02:15] This won't be the first time that script has come in handy [02:16] if you want to DM me a list of urls that it has discovered I can look through it for short URLs if I have spare time [02:16] *** warmwaffl has joined #urlteam [02:16] any little bit helps [02:16] Ok, cool! [02:56] Flashfire: Here are ~4 million URLs pulled from CommonCrawl relating to goo.gl https://drive.google.com/open?id=1tZBCWjtRWT9AWfZIVsTWMliDejTKmfUe [02:57] Ill take a look [03:05] *** Hani has quit IRC (Read error: Connection reset by peer) [03:07] *** Fusl has quit IRC (Ping timeout: 265 seconds) [03:10] *** KingNerd has quit IRC (efnet.portlane.se se.hub) [03:10] *** kiska has quit IRC (efnet.portlane.se se.hub) [03:10] *** Flashfire has quit IRC (efnet.portlane.se se.hub) [03:10] *** seatsea has quit IRC (efnet.portlane.se se.hub) [03:10] *** driib has quit IRC (efnet.portlane.se se.hub) [03:10] *** tech234a has quit IRC (efnet.portlane.se se.hub) [03:10] *** odemg has quit IRC (efnet.portlane.se se.hub) [03:10] *** Jens has quit IRC (efnet.portlane.se se.hub) [03:11] *** zhongfu has quit IRC (efnet.portlane.se se.hub) [03:11] *** bakJAA has quit IRC (efnet.portlane.se se.hub) [03:11] *** t3 has quit IRC (efnet.portlane.se se.hub) [03:11] *** jornbaer has quit IRC (efnet.portlane.se se.hub) [03:11] *** Jusque has quit IRC (efnet.portlane.se se.hub) [03:11] *** mr_archiv has quit IRC (efnet.portlane.se se.hub) [03:11] *** abstract has quit IRC (efnet.portlane.se se.hub) [03:11] *** phuzion has quit IRC (efnet.portlane.se se.hub) [03:11] *** N4Y has quit IRC (efnet.portlane.se se.hub) [03:11] *** gandalf has quit IRC (efnet.portlane.se se.hub) [03:11] *** hook54321 has quit IRC (efnet.portlane.se se.hub) [03:11] *** horkermon has quit IRC (efnet.portlane.se se.hub) [03:11] *** Fusl_ has quit IRC (efnet.portlane.se se.hub) [03:11] *** chr1sm has quit IRC (efnet.portlane.se se.hub) [03:11] *** HCross has quit IRC (efnet.portlane.se se.hub) [03:11] *** pnJay has quit IRC (efnet.portlane.se se.hub) [03:11] *** diggan has quit IRC (efnet.portlane.se se.hub) [03:11] *** JSharp has quit IRC (efnet.portlane.se se.hub) [03:11] *** Ctrl-S_ has quit IRC (efnet.portlane.se se.hub) [03:11] *** deathy has quit IRC (efnet.portlane.se se.hub) [03:11] *** kpcyrd has quit IRC (efnet.portlane.se se.hub) [03:11] *** Hecatz has quit IRC (efnet.portlane.se se.hub) [03:11] *** Muad-Dib has quit IRC (efnet.portlane.se se.hub) [03:11] *** nyany has quit IRC (efnet.portlane.se se.hub) [03:11] *** svchfoo3 has quit IRC (efnet.portlane.se se.hub) [03:11] *** MrRadar2 has quit IRC (efnet.portlane.se se.hub) [03:11] *** treora has quit IRC (efnet.portlane.se se.hub) [03:11] *** BnAboyZ has quit IRC (efnet.portlane.se se.hub) [03:11] *** Dallas has quit IRC (efnet.portlane.se se.hub) [03:12] *** eythian has quit IRC (efnet.portlane.se se.hub) [03:12] *** pikami has quit IRC (efnet.portlane.se se.hub) [03:12] *** warmwaffl has quit IRC (efnet.portlane.se se.hub) [03:12] *** Smiley has quit IRC (efnet.portlane.se se.hub) [03:12] *** Frogging has quit IRC (efnet.portlane.se se.hub) [03:12] *** ivan has quit IRC (efnet.portlane.se se.hub) [03:12] *** JAA has quit IRC (efnet.portlane.se se.hub) [03:12] *** svchfoo1 has quit IRC (efnet.portlane.se se.hub) [03:12] *** lunik1 has quit IRC (efnet.portlane.se se.hub) [03:12] *** marked has quit IRC (efnet.portlane.se se.hub) [03:12] *** hiiva has quit IRC (efnet.portlane.se se.hub) [03:12] *** VADemon_ has quit IRC (efnet.portlane.se se.hub) [03:12] *** Kagee has quit IRC (efnet.portlane.se se.hub) [03:12] *** yano has quit IRC (efnet.portlane.se se.hub) [03:12] *** MrRadar has quit IRC (efnet.portlane.se se.hub) [03:12] *** Somebody2 has quit IRC (efnet.portlane.se se.hub) [03:12] *** arkiver has quit IRC (efnet.portlane.se se.hub) [03:12] *** joepie91 has quit IRC (efnet.portlane.se se.hub) [03:12] *** swebb has quit IRC (efnet.portlane.se se.hub) [03:12] *** chfoo has quit IRC (efnet.portlane.se se.hub) [03:12] *** Cameron_D has quit IRC (efnet.portlane.se se.hub) [03:12] *** Soulflare has quit IRC (efnet.portlane.se se.hub) [03:12] *** Matthww_ has quit IRC (efnet.portlane.se se.hub) [03:12] *** mtntmnky has quit IRC (efnet.portlane.se se.hub) [03:12] *** TigerbotH has quit IRC (efnet.portlane.se se.hub) [03:12] *** Mayonaise has quit IRC (efnet.portlane.se se.hub) [03:12] *** kiska1 has quit IRC (efnet.portlane.se se.hub) [03:13] *** Kaz has quit IRC (Ping timeout: 265 seconds) [03:15] *** Kaz has joined #urlteam [03:17] *** kiskabak has quit IRC (Ping timeout: 265 seconds) [03:17] *** kiskabak has joined #urlteam [03:17] *** Fusl__ has joined #urlteam [03:17] *** Hani has joined #urlteam [03:17] *** warmwaffl has joined #urlteam [03:17] *** tech234a has joined #urlteam [03:17] *** Smiley has joined #urlteam [03:17] *** VADemon_ has joined #urlteam [03:17] *** odemg has joined #urlteam [03:17] *** Jens has joined #urlteam [03:17] *** MrRadar2 has joined #urlteam [03:17] *** nyany has joined #urlteam [03:17] *** seatsea has joined #urlteam [03:17] *** treora has joined #urlteam [03:17] *** BnAboyZ has joined #urlteam [03:17] *** Dallas has joined #urlteam [03:17] *** mtntmnky has joined #urlteam [03:17] *** TigerbotH has joined #urlteam [03:17] *** Mayonaise has joined #urlteam [03:17] *** eythian has joined #urlteam [03:17] *** KingNerd has joined #urlteam [03:17] *** kiska has joined #urlteam [03:17] *** Flashfire has joined #urlteam [03:17] *** Frogging has joined #urlteam [03:17] *** Matthww_ has joined #urlteam [03:17] *** zhongfu has joined #urlteam [03:17] *** marked has joined #urlteam [03:17] *** kiska1 has joined #urlteam [03:17] *** Kagee has joined #urlteam [03:17] *** ivan has joined #urlteam [03:17] *** bakJAA has joined #urlteam [03:17] *** svchfoo3 has joined #urlteam [03:17] *** svchfoo1 has joined #urlteam [03:17] *** se.hub sets mode: +oooo Flashfire bakJAA svchfoo3 svchfoo1 [03:17] *** pikami has joined #urlteam [03:17] *** yano has joined #urlteam [03:17] *** t3 has joined #urlteam [03:17] *** JAA has joined #urlteam [03:17] *** lunik1 has joined #urlteam [03:17] *** hiiva has joined #urlteam [03:17] *** MrRadar has joined #urlteam [03:17] *** Somebody2 has joined #urlteam [03:17] *** arkiver has joined #urlteam [03:17] *** joepie91 has joined #urlteam [03:17] *** swebb has joined #urlteam [03:17] *** jornbaer has joined #urlteam [03:17] *** Jusque has joined #urlteam [03:17] *** mr_archiv has joined #urlteam [03:17] *** abstract has joined #urlteam [03:17] *** se.hub sets mode: +ooo JAA Somebody2 arkiver [03:17] *** driib has joined #urlteam [03:17] *** phuzion has joined #urlteam [03:17] *** N4Y has joined #urlteam [03:17] *** Muad-Dib has joined #urlteam [03:17] *** Hecatz has joined #urlteam [03:17] *** kpcyrd has joined #urlteam [03:17] *** deathy has joined #urlteam [03:17] *** Ctrl-S_ has joined #urlteam [03:17] *** HCross has joined #urlteam [03:17] *** chr1sm has joined #urlteam [03:17] *** diggan has joined #urlteam [03:17] *** pnJay has joined #urlteam [03:17] *** JSharp has joined #urlteam [03:17] *** Fusl_ has joined #urlteam [03:17] *** horkermon has joined #urlteam [03:17] *** hook54321 has joined #urlteam [03:17] *** gandalf has joined #urlteam [03:17] *** Soulflare has joined #urlteam [03:17] *** Cameron_D has joined #urlteam [03:17] *** chfoo has joined #urlteam [03:17] *** se.hub sets mode: +ooo HCross hook54321 chfoo [03:20] Flashfire: ~3.7 million bit.ly URLs https://drive.google.com/open?id=1ozXk6mx8lBw5IVLnLxmdoIN1Hu3qDLKz [03:21] Bitly we currently have running through the warriors but thats still useful to look through [03:22] I'll see what I can pull for TinyURL [03:23] See if you have anything for stuff that hasnt been proccessed through the warrior [03:23] Such as? [03:23] https://www.archiveteam.org/index.php?title=URLTeam#Alive [03:23] anything from there that isnt also in https://www.archiveteam.org/index.php?title=URLTeam#Warrior_projects [03:24] Ok [03:24] for example cl.ly is in the alive section but not the warrior section [03:29] *** odemg has quit IRC (Ping timeout: 615 seconds) [03:30] *** Flashfire sets mode: +o Kaz [03:36] *** odemg has joined #urlteam [03:36] Flashfire: ~900k tinyurl.com URLs https://drive.google.com/open?id=1LUCQUZamqDlq0uTQe9ybLQHEWZmLp5Ol [03:36] cl.ly next [03:37] WOW [03:43] Hmm... my script is acting slightly funny with cl.ly, probably because of the low number of URLs that were discovered. Looking into it... [03:59] Flashfire: So it wasn't able to retrieve anything for cl.ly even though a few URLs showed up in the logs. If you can get me a comma or new-line separated list of domains to check, I can run it tomorrow. [04:00] (There is a small bug in the script where it might drop up to 56 URLs per service.) [04:08] *** VADemon__ has joined #urlteam [04:12] *** VADemon_ has quit IRC (Read error: Operation timed out) [04:14] *** Fusl__ is now known as Fusl [04:24] *** warmwaffl has quit IRC (Remote host closed the connection) [04:27] ok [06:01] *** DustinV has joined #urlteam [06:34] *** DustinV has quit IRC (Read error: Operation timed out) [07:15] http://accntu.re a bitly alias [08:07] *** tech234a has quit IRC (Quit: Connection closed for inactivity) [08:15] *** Zerote has joined #urlteam [12:23] *** tech234a has joined #urlteam [12:44] *** flugga has joined #urlteam [12:51] *** flugga has quit IRC (Quit: Page closed) [13:03] *** morgan_ has joined #urlteam [14:31] *** Zerote has quit IRC (Ping timeout: 260 seconds) [14:33] *** tech234a has quit IRC (Quit: Connection closed for inactivity) [14:56] *** tech234a has joined #urlteam [16:12] *** VADemon__ is now known as VADemon [16:13] Flashfire: cyrillic and alphabetical sorting WHERE? What language or what purpose? I am a native русский speaker, auto-qualified for the job :) [16:14] *** kiska1 has quit IRC (Read error: Operation timed out) [16:15] *** kiska1 has joined #urlteam [16:21] *** kiska1 has quit IRC (Ping timeout (120 seconds)) [16:21] *** kiska1 has joined #urlteam [16:41] *** syn_ has joined #urlteam [16:42] *** syn_ has quit IRC (Client Quit) [17:20] *** Zerote has joined #urlteam [17:23] *** tech234a has quit IRC (Quit: Connection closed for inactivity) [17:43] *** warrior88 has joined #urlteam [17:47] hello [17:47] I'm getting "Error communicating with tracker: 507 Server Error: The tracker needs an operator for manual maintenance. Try again later. for url: https://tracker.archiveteam.org:1338/api/get." errors [17:48] I'm hoping someone here can help with that [17:48] I'm pretty sure it's a server side issue [17:49] *** Veeb0rg has joined #urlteam [17:50] hey. new here got a issue/question.. [17:51] keep getting Error communicating with tracker: 507 Server Error: The tracker needs an operator for manual maintenance. Try again later. for url: https://tracker.archiveteam.org:1338/api/get. [17:53] Veeb0rg: same for me [17:53] I believe it's a server side problem [17:54] guess the other question should be is there any advantage in increasing the core and memory available to the vm? [17:54] core count [18:56] *** Zerote has quit IRC (Ping timeout: 260 seconds) [19:08] *** JackTerok has joined #urlteam [19:09] Hi guys. Just found this project today and im trying to set it up. Im getting Error communicating with tracker: 507 Server Error: The tracker needs an operator for manual maintenance. Try again later. for url: https://tracker.archiveteam.org:1338/api/get. [19:10] Is that a problem on my end? [19:13] *** tech234a has joined #urlteam [19:14] Probably not. I'm having the same error [19:15] 4x independent reports of 507 errors now [19:15] Ok Thx [19:16] And the tracker says "0 scans per second" [19:16] https://tracker.archiveteam.org:1338/ [19:26] Looking into it. [19:28] Somebody2: dlvr-it is throwing a lot of "terroroftinytown.client.errors.UnexpectedNoResult: Unexpectedly did not get a body result" errors. [19:29] Disabled dlvr-it auto-queue and cleared the errors. [19:29] Its working \o/ [19:34] *** Zerote has joined #urlteam [20:03] *** JackTerok has quit IRC (Quit: Page closed) [20:13] *** SmileyG has joined #urlteam [20:14] *** Smiley has quit IRC (Read error: Operation timed out) [20:22] Is there any advantage to giving the VM more cpu cores or ram? [20:22] Veeb0rg: For URLTeam, no. This project uses near-zero resources. [20:23] oh, well i set it to 16cores and 16gb ram anyway. since there is nothing else running on the box anyway [20:57] *** Veeb0rg has quit IRC (Quit: Page closed) [21:42] *** tech234a has quit IRC (Quit: Connection closed for inactivity) [21:58] you dont even need more than 2-4 cores and 2GB of ram in any way you would use it ^_^ [22:13] *** warrior88 has quit IRC (Ping timeout: 260 seconds) [23:15] *** Freiner has joined #urlteam [23:22] *** tech234a has joined #urlteam [23:23] *** Freiner has quit IRC (Quit: Page closed) [23:49] *** Zerote has quit IRC (Ping timeout: 260 seconds)