#archiveteam-ot 2018-09-28,Fri

↑back Search

Time Nickname Message
02:28 🔗 logchfoo0 starts logging #archiveteam-ot at Fri Sep 28 02:28:59 2018
02:28 🔗 logchfoo0 has joined #archiveteam-ot
02:36 🔗 BlueMax has joined #archiveteam-ot
02:43 🔗 erin has quit IRC (Ping timeout: 264 seconds)
02:49 🔗 zino has quit IRC (Excess Flood)
02:52 🔗 zino has joined #archiveteam-ot
03:10 🔗 erin has joined #archiveteam-ot
03:14 🔗 JAA has joined #archiveteam-ot
03:14 🔗 svchfoo3 sets mode: +o JAA
03:15 🔗 bakJAA sets mode: +o JAA
03:18 🔗 jspiros has joined #archiveteam-ot
03:43 🔗 odemg has quit IRC (Ping timeout: 260 seconds)
03:55 🔗 odemg has joined #archiveteam-ot
04:14 🔗 Frogging has quit IRC (Ping timeout: 268 seconds)
04:16 🔗 Frogging has joined #archiveteam-ot
04:21 🔗 Mateon1 has quit IRC (Read error: Connection reset by peer)
04:21 🔗 Mateon1 has joined #archiveteam-ot
04:24 🔗 BnAboyZ has joined #archiveteam-ot
04:27 🔗 VoynichCr has quit IRC (Ping timeout: 268 seconds)
04:27 🔗 VoynichCr has joined #archiveteam-ot
04:29 🔗 BnAboyZ has quit IRC (Ping timeout: 268 seconds)
04:30 🔗 BnAboyZ has joined #archiveteam-ot
07:48 🔗 ivan update grab-site for quarantined subreddit support
07:52 🔗 ivan my snscrape is running and I'm going to archive a lot of twitter users, send me yours if you want them archived
07:54 🔗 BlueMax has quit IRC (Read error: Connection reset by peer)
07:55 🔗 BlueMax has joined #archiveteam-ot
08:01 🔗 dxrt ivan: Could you please do twitter user morpheus______ for me. Thanks
08:02 🔗 Flashfire Was I doing this wrong I sent dms to ivan
08:02 🔗 hook54321 JAA: On snscrape, is using \ before spaces when doing twitter searches correct?
08:03 🔗 hook54321 Also, is the "quality filter" on or off by default?
08:06 🔗 JAA hook54321: That's not really an issue with snscrape but more with your shell. snscrape just needs to see it as a single argument. In typical shells, both snscrape twitter-search a\ search and snscrape twitter-search 'a search' (and a number of other variations) are fine.
08:06 🔗 hook54321 k
08:06 🔗 ivan thanks, I will process all of the users mentioned at me in a bit
08:07 🔗 JAA snscrape currently doesn't disable the quality filter. It's something I only came across a few days ago. I'll add it this weekend.
08:09 🔗 Mateon1 has quit IRC (Ping timeout: 260 seconds)
08:09 🔗 Mateon1 has joined #archiveteam-ot
08:19 🔗 ivan does the quality filter affect from: searches?
08:51 🔗 ivan I'm testing QFD-banned users on https://shadowban.eu/ and they're no longer marked as QFD-banned
08:52 🔗 ivan they seem to show up in search too when using from:
08:55 🔗 ivan snscrape isn't going to find retweets for me, is it
08:56 🔗 ivan https://twitter.com/starryneutrons/status/1045073527303417856 is showing up in search, for example, I guess something changed in the last few days
09:10 🔗 Flashfire Ivan snscrape doesnt grab retweets
09:37 🔗 * ivan imports https://twitter.com/cspan/lists/members-of-congress/members?lang=en
09:56 🔗 VerifiedJ has joined #archiveteam-ot
10:54 🔗 ivan scraping 2787 accounts with snscrape
11:06 🔗 ivan does anyone have a list of all bluechecks somewhere
11:22 🔗 Flashfire Ivan let me know when you are done I will find some more
11:31 🔗 JAA ivan: Yeah, because of how snscrape discovers tweets (through the search page rather than the user profile, because the latter is limited to 3200 results while the former is not), it can't discover retweets. Or at least I haven't found any way to do that. The search term suggestions I found online are all several years old and no longer work.
11:35 🔗 JAA hook54321: FYI, I filed the quality filter issue as https://github.com/JustAnotherArchivist/snscrape/issues/3
11:41 🔗 hook54321 ok
11:54 🔗 JAA Also created issues for a bunch of other things. More to come later.
11:58 🔗 ivan does anyone have a kv-store-backed thing for incrementally loading URLs into giant database and producing N-sized items
11:58 🔗 ivan I looked at my greader-item-maker from 2013 and it's a hot mess
12:02 🔗 ivan might finally have to excuse to write some Rust again
12:14 🔗 ivan online.net sent me an email, prices are going up for existing servers
12:23 🔗 fenn redis?
12:38 🔗 ivan I was going to use a rocksdb
13:05 🔗 ivan TIL twitter doesn't need a valid username in the URL, it will redirect if the tweet id is good
13:13 🔗 Mateon1 has quit IRC (Remote host closed the connection)
13:14 🔗 Mateon1 has joined #archiveteam-ot
13:28 🔗 VerifiedJ has quit IRC (Read error: Operation timed out)
14:49 🔗 JAA Added a bunch more issues for a variety of things.
14:49 🔗 JAA (In case anyone's interested in what I have in mind to do with snscrape soonish.)
15:00 🔗 VerifiedJ has joined #archiveteam-ot
15:05 🔗 BlueMax has quit IRC (Quit: Leaving)
15:20 🔗 wp494 has quit IRC (Read error: Operation timed out)
15:21 🔗 wp494 has joined #archiveteam-ot
16:41 🔗 godane has quit IRC (Read error: Operation timed out)
17:24 🔗 godane has joined #archiveteam-ot
17:24 🔗 svchfoo1 sets mode: +o godane
18:14 🔗 bithippo has joined #archiveteam-ot
18:46 🔗 ColdIce Uhm, how do I contribute my WARC?
20:55 🔗 ivan ColdIce: if the site is still up and you need it in wayback, maybe rearchive with ArchiveBot
20:56 🔗 ivan otherwise upload to an IA item but it will probably not get ingested to wayback
21:07 🔗 ColdIce site is still up
21:07 🔗 ColdIce it's a site for terminal emulator on TIKI-100
21:07 🔗 robogoat has quit IRC (Read error: Operation timed out)
21:08 🔗 ivan ColdIce: /join #archivebot
21:09 🔗 astrid has left ][
21:11 🔗 robogoat has joined #archiveteam-ot
23:42 🔗 BlueMax has joined #archiveteam-ot

irclogger-viewer