[00:00] *** pizzaiolo has quit IRC (Ping timeout: 506 seconds) [00:00] *** pizzaiolo has joined #archiveteam-bs [00:02] *** pizzaiolo has quit IRC (Client Quit) [00:03] *** pizzaiolo has joined #archiveteam-bs [00:36] xmc: I have thought about making a botnet that is if you are an op in here you can control it, have it join channels and op you in ever channel you and the bots are in upon request [00:36] basic commands like !join #channel and !opme -- with a small whitelist of people who have static hostnames in case of failure to auto op [00:39] have 3 bots connected to server that is connected to a different hub [00:42] *** pnJay has quit IRC (Leaving) [00:55] *** BlueMaxim has joined #archiveteam-bs [01:06] *** dxrt- has joined #archiveteam-bs [01:14] *** NstkVdwn has quit IRC (Quit: Leaving) [01:17] *** th1x has joined #archiveteam-bs [01:36] *** JensRex has joined #archiveteam-bs [01:37] *** username1 has joined #archiveteam-bs [01:38] *** j08nY has quit IRC (Quit: Leaving) [01:39] *** schbirid2 has quit IRC (Read error: Operation timed out) [02:19] *** wm_ has quit IRC (Ping timeout: 260 seconds) [02:20] *** wm_ has joined #archiveteam-bs [02:22] *** Fusl has quit IRC (Ping timeout: 250 seconds) [02:34] *** Fusl has joined #archiveteam-bs [03:13] *** qw3rty has joined #archiveteam-bs [03:18] *** qw3rty2 has quit IRC (Read error: Operation timed out) [03:30] *** pizzaiolo has quit IRC (Quit: pizzaiolo) [03:47] *** robink has quit IRC (Ping timeout: 246 seconds) [03:48] *** robink has joined #archiveteam-bs [03:59] *** icedice has quit IRC (Ping timeout: 245 seconds) [04:07] *** jspiros has quit IRC (Ping timeout: 492 seconds) [04:54] *** Sk1d has quit IRC (Ping timeout: 194 seconds) [05:00] *** Sk1d has joined #archiveteam-bs [05:14] *** jspiros has joined #archiveteam-bs [05:18] *** th1x has quit IRC (Read error: Operation timed out) [05:35] *** th1x has joined #archiveteam-bs [06:41] *** username1 has quit IRC (Quit: Leaving) [06:45] *** schbirid has joined #archiveteam-bs [06:46] *** Honno has joined #archiveteam-bs [06:51] *** kimmer has quit IRC (Read error: Connection reset by peer) [06:52] *** kimmer has joined #archiveteam-bs [07:47] *** kimmer has quit IRC (Read error: Connection reset by peer) [08:01] *** kimmer22 has joined #archiveteam-bs [08:09] *** kimmer2 has quit IRC (Ping timeout: 632 seconds) [08:15] *** kimmer2 has joined #archiveteam-bs [08:20] *** th1x has quit IRC (Read error: Operation timed out) [08:23] *** kimmer22 has quit IRC (Ping timeout: 633 seconds) [09:16] *** icedice has joined #archiveteam-bs [09:22] *** schbirid2 has joined #archiveteam-bs [09:26] *** schbirid has quit IRC (Read error: Operation timed out) [09:44] *** Honno has quit IRC (Read error: Operation timed out) [10:10] *** icedice has quit IRC (Quit: Leaving) [10:21] *** kimmer22 has joined #archiveteam-bs [10:27] *** kimmer2 has quit IRC (Read error: Operation timed out) [10:32] *** kimmer2 has joined #archiveteam-bs [10:36] *** kimmer23 has joined #archiveteam-bs [10:39] *** icedice has joined #archiveteam-bs [10:42] *** kimmer24 has joined #archiveteam-bs [10:42] *** kimmer22 has quit IRC (Ping timeout: 633 seconds) [10:43] *** kimmer2 has quit IRC (Ping timeout: 633 seconds) [10:44] *** kimmer has joined #archiveteam-bs [10:51] *** username1 has joined #archiveteam-bs [10:52] *** kimmer2 has joined #archiveteam-bs [10:52] *** kimmer23 has quit IRC (Ping timeout: 633 seconds) [10:55] *** schbirid2 has quit IRC (Read error: Operation timed out) [10:58] *** kimmer24 has quit IRC (Ping timeout: 633 seconds) [11:02] *** icedice has quit IRC (Quit: Leaving) [11:09] *** kimmer22 has joined #archiveteam-bs [11:18] *** kimmer2 has quit IRC (Read error: Operation timed out) [11:19] *** kimmer22 has quit IRC (Read error: Connection reset by peer) [11:20] *** kimmer2 has joined #archiveteam-bs [11:37] *** kimmer22 has joined #archiveteam-bs [11:39] *** j08nY has joined #archiveteam-bs [11:43] *** quantum has joined #archiveteam-bs [11:47] *** kimmer2 has quit IRC (Ping timeout: 633 seconds) [12:18] *** pizzaiolo has joined #archiveteam-bs [12:23] *** pnJay has joined #archiveteam-bs [12:27] *** kimmer22 has quit IRC (Ping timeout: 633 seconds) [12:28] *** pizzaiolo has quit IRC (pizzaiolo) [12:31] *** pizzaiolo has joined #archiveteam-bs [12:33] *** quantum has quit IRC (Quit: Page closed) [12:40] *** dxrt sets mode: +o dxrt- [12:46] *** th1x has joined #archiveteam-bs [13:28] *** BlueMaxim has quit IRC (Quit: Leaving) [13:35] *** schbirid2 has joined #archiveteam-bs [13:38] *** username1 has quit IRC (Read error: Operation timed out) [13:38] *** bwn has quit IRC (Ping timeout: 268 seconds) [13:49] *** NstkVdwn has joined #archiveteam-bs [13:57] *** pizzaiolo has quit IRC (Quit: pizzaiolo) [13:58] *** pizzaiolo has joined #archiveteam-bs [14:01] *** pizzaiolo has quit IRC (Client Quit) [14:01] *** pizzaiolo has joined #archiveteam-bs [14:41] *** TC01 has quit IRC (Read error: Operation timed out) [14:42] *** TC01 has joined #archiveteam-bs [15:02] *** bwn has joined #archiveteam-bs [15:30] *** bwn has quit IRC (Read error: Operation timed out) [15:30] *** bwn_ has joined #archiveteam-bs [15:30] *** bwn_ is now known as bwn [16:04] *** NstkVdwn has quit IRC (Ping timeout: 506 seconds) [16:12] *** NstkVdwn has joined #archiveteam-bs [17:10] *** tsuckow has joined #archiveteam-bs [17:14] tsuckow: what you having issues with [17:15] arm is a PITA, but if you are just dedicating a machine to it [17:15] just run the commands inside the dockerfile by hand [17:16] jrwr: At the moment building wget-lua. Nothing blocking, just keep finding packages I need to install which takes forever. [17:16] ya [17:16] the docker file has the apt-get line that covers most of it [17:17] I switched to raspbian base image because it is armv6 but it apparently doesn't include some packages by default like the other one [17:17] *** fallenoak has joined #archiveteam-bs [17:17] and the prebuilt wget-lua isn't compatible [17:21] Some day I will finish making a backup utility for C.H.I.P. and I could start using those. [17:21] I just got done making a new warriorvm [17:21] the current ones we use are kind of old [17:21] like 2010 old [17:22] If it's not broke take it apart and find out why. [17:22] oh but it is [17:22] the SSL engine in that thing is so old [17:23] some modern websites just fail to work [17:23] Ya, I noticed python bitching [17:24] it uses Alpine Linux and Docker [17:25] it just uses the docker version (so it can stay up to date) on boot [17:25] 60MB [17:26] nearly a third smaller. [17:27] it downloads a 300MB Docker image [17:27] so it comes out in the wash [17:27] It looked like you also upped the disk to 100GB. Do the projects ever really approach the 60GB? [17:27] some can [17:28] I've seen some come back before [17:28] but if its not used, its not used [17:29] https://archive.org/download/AT-Warrior100G/Warrior-100G.ova Your welcome to try it, its "Unsupported" but ill help where I can [17:32] One of these days I need to look at just running the docker image on windows 10 [17:33] Though the point of running it on the pi is so I can turn the desktop off. [17:35] Vm seems to work fine [17:35] Nice :) [17:37] If you wanted to be as minimal as the old one you could disable usb and reduce video memory to 1MB [17:38] But i doubt it matters [17:38] Save all of 10MB of ram [17:38] on boot the base os uses 40MB [17:38] then once the warrior boots its 100MB [17:38] so [17:39] if you switch to TTY2 its root:warrior [17:39] htop is installed [17:41] You must know what yuo are doing if you put htop init [17:42] lol [17:43] or, we don't know what it's doing :) [17:43] lol [17:43] its a base alpine install with a boot.sh as TTY1 that just runs docker and checks if docker is running [17:47] I am going to write some docs on the edits I did (/etc/inittab) and apk add docker htop nano [17:51] Anyone know why isc-dhcp-client is in the docker container? [18:06] Is there a faster way to archive a site rather than just doing wget -m? [18:14] wpull -m with concurrency and all that shit [18:14] oh didn't know wpull did concurrency [18:19] *** phuzion has joined #archiveteam-bs [18:30] *** TheLovina has joined #archiveteam-bs [18:37] *** Odd0002 has quit IRC (Remote host closed the connection) [19:13] *** Asparagir has joined #archiveteam-bs [19:21] *** NstkVdwn has quit IRC (Quit: Leaving) [19:25] *** Odd0002 has joined #archiveteam-bs [19:28] *** username1 has joined #archiveteam-bs [19:34] *** schbirid2 has quit IRC (Read error: Operation timed out) [19:50] *** kimmer has quit IRC (Read error: Connection reset by peer) [19:51] *** kimmer has joined #archiveteam-bs [19:51] *** username1 has quit IRC (Quit: Leaving) [19:53] *** Odd0002 has quit IRC (Remote host closed the connection) [20:00] *** Honno has joined #archiveteam-bs [20:16] *** ja0Hai has joined #archiveteam-bs [20:24] *** th1x has quit IRC (Leaving) [20:24] *** th1x has joined #archiveteam-bs [20:34] *** schbirid has joined #archiveteam-bs [20:55] *** schbirid2 has joined #archiveteam-bs [20:56] *** schbirid has quit IRC (Read error: Operation timed out) [20:58] *** schbirid2 has quit IRC (Remote host closed the connection) [21:17] *** Sk1d has quit IRC (Ping timeout: 250 seconds) [21:17] *** pnJay has quit IRC (Quit: Leaving) [21:21] *** Sk1d has joined #archiveteam-bs [21:21] *** Sk1d has quit IRC (Connection Closed) [21:21] *** Sk1d has joined #archiveteam-bs [21:21] Fyi, FamilySearch is discontinuing microfilm/fiche distribution on September 1, 2017. If you want something, send it to a Family History Center by August 31. [21:21] * Asparagir waves to fellow genealogy nerd [21:21] * hook54321 waves back [21:22] Also, if you know any genealogical/archival records that you want to see released to the public, send my org a heads up: www.ReclaimTheRecords.org [21:22] *** ndiddy has quit IRC (Read error: Operation timed out) [21:22] We use Freedom of Information laws to force government archives, libraries, and agencies to hand over copies of data they have. [21:22] And if they don't comply, we sue. :-) [21:23] Lots of success in the past two years, millions of records uploaded to the Internet Archive. And we just filed for non-proft status with the IRS so my little project is no going big-time. [21:23] They're going to apparantly have it all digitized by 2020. But I'm not sure how or if copyright will affect what they have publicly available online. [21:23] *now [21:24] What kind of records would FOI laws require them to hand over? [21:24] It will, because some countries (Ukraine and Poland, for example) need to have their contracts renegotiated to allow the microfilm images to go online, not just be on film. Some of their books were filmed in the 1980's before online images existed. [21:25] Re: FOI -- In the past two years, I got the NYC marriage index 1908-1929 from the NYC Municipal Archives, then followed that up with getting the rest of teh record set 1930-1995 from the New York City Clerk's Office. [21:25] Neither had EVER been available online before, nor on FamilySearch microfilm. [21:25] Because NYC are jerks about not allowing access. [21:26] Speaking of which, NYC is another example where they have b/m/d certificates on microfilm, but they are refusing to allow FamilySearch to put the images online! So FS went ahead and did a text-transcription of the images. Which is not quite as good. [21:27] Unfortunately (at least for me) it costs $7.50 for shipping and handling to get an item delivered to a Family History Center. [21:27] Yeah. [21:27] I also got the New York State (minus NYC) death index 1880-1956 through an FOI fight over the past two years and just finished uploading it all to the IA. No one had that! NY State! [21:27] It's amazing what FOI laws and the willingness to fight can do. :-) [21:27] Got lots of other stuff too. [21:28] And have a lawsuit pending in MIssouri for the first ever copy of their birth index (post-1910) and death index (post-1966). Very very basic index, not actual certificates, but they're being jerks and don't want to hand them over. Even though they sell that same data! Or maybe because they sell thaht same data, and don't want to lose the revenue stream. [21:28] hmm... If someone lives in Utah, could they just walk into the Family History Library and start making digital copies of the NYC d/m//d certificates? [21:29] Yup! From the films. [21:29] But NYC is refusing to grant the right to put the images online. [21:29] Why doesn't NYC let them put it online? [21:29] I mean, there's only so much they can do to keep it offline... [21:29] Because they want to be the sole source of this data. They're data hoarders. They make money selling copies. [21:29] It's like Gollum with the ring MY PRECCCCIOUS [21:30] So I have no compunction about suing them (twice now!) to get data from them. [21:30] But I haven't gone after any actual certificates yet. [21:30] Getting an index under FOI laws seems much easier. [21:30] Getting a certificate will be a harder fight. [21:31] Asparagir: Are you a lawyer? [21:31] Or is it easy enough to sue under FOIA that you don't need one? [21:31] Couldn't someone get digital copies at the Family History Library in Utah and then host it on a server in another country? [21:31] Not a lawyer, but my parents wish I were. :-) [21:31] I was pre-law in college. [21:31] To my lay mind, I'd have thought suing the government would be expensive and time consuming haha [21:32] I had/have attorneys for all three lawsuits. [21:32] Suing State And Federal Government For Fun And Profit [21:32] Awesome [21:32] What would happen if someone uploaded the NYC b/m/d certificates to libgen? [21:32] #1 was against the NYC Municipal Archives' parent agency DORIS. I won settlement and got all records, but did not win my attorneys fees. Luckily they were not bad at all, because I used a public interest law firm that likes to stick up for the litle guys. [21:33] Lawuist #2 was against the New York City Clerk's Office. I won a settlement again, won all the records but had to agree to take slightly redacted copies (which was okay). And that time I did win attorneys fees! [21:34] why did they redact some stuff? [21:34] The thing is, for STATE FOI requests, most states allow you to potentailly win your attorneys fees, but only five state mandate that you will definitely get reimbursed if you win the records: NJ, California, and three more I don't remember right now. [21:35] *** pnJay has joined #archiveteam-bs [21:35] Also, what are we going to do about this? : http://www.thedailybeast.com/cia-plans-to-destroy-some-of-its-old-leak-files [21:35] They redacted the bride and groom's dates of birth. They claimed that part was too invasive. I asked if they could just leave in the year of birth and cut the month/day, but they said no. It's an unsettled part of NY FOIL (their FOI law) whether dates of birth need to be redacted or not. So I could have fought them in court over that issue. But I decided to just take the rest of the data instead. [21:38] They also cut the upper bound of the years off at 1995. That's because my FOIL request had asked them for a copy of the marriage index. But it turns out that starting in 1996, there isn't any separate index for NYC marriages, the data was "born digital" in database form right at the city clerk's office window. [21:38] So I need to file a new FOI request later this year asking for "a redcated section of the NYC marriage database" to get the rest of the years, 1996-2016. It's an index but I can't call it an index in my request, what a pain. [21:39] why do we want all this stuff online anyway? [21:39] Because genealogists like having open records? [21:40] Because public records belong to the public and we're tired of getting gouged on $22 fees per record search. [21:40] I have a 503 MB text file titled "voters" [21:40] idk why birth certificates are public records anyway [21:40] The certificates usually are not. The INDEX to the sometimes is. [21:40] *them [21:40] You don't want to enable identity fraud. [21:40] Does anyone want this text document? [21:41] What's in it? [21:41] exactly that's what I was thinking [21:41] ASCII art [21:41] of "voters" repeated 300 million times [21:41] Notepad is having issues opening [21:41] a bunch of porn in base64 [21:41] it [21:41] Probably because it's 503 MB [21:41] use less [21:41] :p [21:42] It's something that was leaked like a year a ago. [21:42] or head or tail or cat or anything that doesn't try and load the whole file into memory at once [21:43] then it's probably https://www.forbes.com/sites/thomasbrewster/2015/12/28/us-voter-database-leak/#491dd53c5b98 [21:43] so, nothx [21:43] and display it in a win32 text control [21:44] Top of the text document: "Registration Date","Original Registration Date","Party","Phone","Mailing Address","Mailing city, state zip","County ID","Precinct","House Number","House Number Suffix","Direction Prefix","Street","Direction Suffix","Street Type","Unit Type","Unit Number","City","Zip","DOB","Congressional","State House","State Senate","State Schoolboard","Local Schoolboard","County Council","City Council" [21:44] FYI, lots of states (like NY) allow you to get copies of their voter databases under their FOI laws! They just want you to swear you won't use the addresses to sign people up for junk mail. The laws vary a lot from state to state. [21:45] this is also on the top line: [21:45] "11/6/1990","11/5/1991","11/3/1992","11/2/1993","11/8/1994","5/23/1995","9/12/1995","10/3/1995","11/7/1995","6/25/1996","8/6/1996","11/5/1996","2/4/1997","5/6/1997","8/1/1997","10/7/1997","11/4/1997","6/23/1998","11/3/1998","5/4/1999","8/3/1999","10/5/1999","11/2/1999","5/2/2000","6/27/2000","11/7/2000","2/6/2001","10/2/2001","11/6/2001","6/25/2002","11/5/2002","2/4/2003","8/5/2003","10/7/2003","11/4/2003","5/4/2004"," [21:45] 6/22/2004","8/3/2004","11/2/2004","10/4/2005","11/8/2005","6/27/2006","11/7/2006","6/26/2007","9/11/2007","11/6/2007 [21:46] I'm pretty sure this is just people in Utah... [21:46] Yeah. Just people in Utah. [21:46] No idea what it all means though [21:47] oh. You can open it in LibreOffice Calc. [21:47] the unexplored cyberpunk scenario [21:47] "I have all this data, I don't know what to do with it" [21:48] "I just download stuff people send me" [21:48] yipdw: I think that's a different data set though. The one you linked was a ~300 GB MongoDB IIRC. [21:48] LibreOffice now just has a white screen. [21:48] JAA: oh ok, well in that case it's comforting to know that database will have all sorts of integrity problems [21:49] Actually, apparantly I downloaded this in 2013... O_o [21:49] lol [21:49] Never Forget [21:49] To Rag On MongoDB [21:50] At least according to the Created and Accessed dates [21:51] https://raidforums.com/Thread-Utah-Voter-Database-Leaked-Download [21:51] The date on that is 2017 though [21:54] How many records does that file have, hook54321? [21:55] I'll tell you if I can get LibreOffice to open it [21:55] wc -l ftw [21:55] ? [21:55] Or are you on Windows? [21:56] *** ndiddy has joined #archiveteam-bs [21:58] yup. It's on a computer in a Family History Center. [21:58] I see. No clue then. [22:00] It might have been from this site: http://utvoters.com/ [22:06] Yep, that sounds right. 528252166 bytes, created on 2013-06-21 20:42. [22:07] wait, how did you? [22:07] As mentioned on the page you linked, this is still downloadable from https://www.indymedia.org.uk/en/2014/02/515559.html [22:09] Yup. That's the one. Same exact size. [22:09] :-) [22:12] Is there anything we can do about this? http://www.thedailybeast.com/cia-plans-to-destroy-some-of-its-old-leak-files [22:21] you can try FOIAing them, but it will take a while and there's no guarantee [22:23] Asparagir ^ ? [22:25] Emma Best (@NatSecGeek on Twitter) and Nate Jones (@NSANate on twitter) would be your best people to ask. [22:25] Will contacting them get me put on some list? [22:26] *shrugs* [22:26] They're already trying to stop the destruction: https://twitter.com/NSANate/status/887300618217500672 [22:26] Looks like NARA is taking comments from the public at request.schedule@nara.gov . [22:27] They would have tried to FOIAing them if they could have, right? [22:28] *** mhazinsk has quit IRC (Read error: Operation timed out) [22:28] *** mhazinsk has joined #archiveteam-bs [22:29] I guess? Why not ask them? [22:29] list [22:29] Whose list? [22:29] **the** list [22:29] You're already on that one. [22:29] why? [22:29] We're allll on someone's list. Some of us were born there! [22:30] Because you mentioned the list. [22:30] :/ [22:30] *** kimmer2 has joined #archiveteam-bs [22:31] Fub fact: Emma Best, under her previous name Mike Best, is one of the people who has uploaded the most files to the Internet Archive. She's our kind of person -- likes open records, saving history, and so on. Because of her work, CIA finally put their CREST database online earlier this year. [22:31] *Fun [22:31] CREST database? [22:32] It's super-cool! One sec, let me dig up the link... [22:32] https://www.muckrock.com/news/archives/2017/jan/17/cias-declassified-database-now-online/ [22:32] Direct link: https://www.cia.gov/library/readingroom/collection/crest-25-year-program-archive/ [22:33] Emma has already started uploading/archiving/backing up the whole database to the Internet Archive. [22:33] I think it's probably still rendering/deriving. [22:35] Just checked the Catalog page and she's still at it. 1.2 million uploads and counting: https://archive.org/details/@the_mike_best [22:35] I just followed Emma Best on Twitter, right after I did that Twitter recommended that I follow Chelsea Manning 🤔 [22:35] Hahaha. [22:38] *** omglolbah has joined #archiveteam-bs [23:14] Asparagir: that answers so much- I was wondering why I couldn't Mike Best around, because I was going to point out that effort [23:19] *** Atom has joined #archiveteam-bs [23:27] dashcloud: "couldn't __?__ Mike Best around" [23:28] couldn't find the name online [23:28] oh [23:28] https://archive.org/details/@the_mike_best [23:30] thanks