#archiveteam-bs 2020-05-30,Sat

↑back Search

Time Nickname Message
00:11 🔗 katocala has quit IRC (Ping timeout: 255 seconds)
00:11 🔗 katocala has joined #archiveteam-bs
00:26 🔗 BlueMax has joined #archiveteam-bs
00:31 🔗 Arcorann has joined #archiveteam-bs
00:44 🔗 arkiver We now have a new customized wget with full ZSTD compression support
00:45 🔗 arkiver with ZSTD we can use trained custom dictionaries, which overall decrease the size of certain WARC records enormously
00:45 🔗 arkiver a trainer is in place which will train a dictionary on the latest WARC record uploaded to IA for a certain project.
00:46 🔗 arkiver the new dictionary is then used by warrior and workers to compress data with
00:47 🔗 arkiver this can decrease the size of data with tens of percentages overall, which means less outgoing bandwidth, and less disk usage on IA
00:57 🔗 OrIdow6 Does this work in wget-lua, or just plain wget?
00:58 🔗 arkiver it's wget-at, currently at https://github.com/ArchiveTeam/wget-lua/tree/v1.20.3-at and will be merged with master later
00:59 🔗 arkiver it's renamed to wget-at (from wget-lua) since this now includes more than just the Lua scripting option
00:59 🔗 arkiver it also includes the deduplication, URL agnostic deduplication option, and ZSTD compression with dictionaries
01:10 🔗 DLoader_ has joined #archiveteam-bs
01:20 🔗 DLoader has quit IRC (Ping timeout: 745 seconds)
01:20 🔗 DLoader_ is now known as DLoader
01:36 🔗 lennier2 has joined #archiveteam-bs
01:48 🔗 lennier1 has quit IRC (Ping timeout: 745 seconds)
01:48 🔗 lennier2 is now known as lennier1
01:53 🔗 lennier1 has quit IRC (Quit: Going offline, see ya! (www.adiirc.com))
01:54 🔗 Meli has quit IRC (Read error: Operation timed out)
01:55 🔗 Frogging has quit IRC (Read error: Operation timed out)
01:55 🔗 Frogging has joined #archiveteam-bs
01:55 🔗 legoktm has quit IRC (Read error: Operation timed out)
01:56 🔗 legoktm has joined #archiveteam-bs
01:56 🔗 Ctrl has quit IRC (Read error: Operation timed out)
01:57 🔗 Raccoon` has joined #archiveteam-bs
01:58 🔗 wessel152 has quit IRC (Read error: Operation timed out)
02:00 🔗 Raccoon has quit IRC (Read error: Operation timed out)
02:00 🔗 Raccoon` is now known as Raccoon
02:03 🔗 Pixi has quit IRC (Read error: Connection reset by peer)
02:03 🔗 Xibalba has quit IRC (Read error: Operation timed out)
02:03 🔗 Xibalba has joined #archiveteam-bs
02:03 🔗 Pixi has joined #archiveteam-bs
02:03 🔗 Xibalba has quit IRC (Handshake flooding)
02:03 🔗 Xibalba has joined #archiveteam-bs
02:04 🔗 lennier1 has joined #archiveteam-bs
02:04 🔗 phuzion_ has joined #archiveteam-bs
02:11 🔗 jodizzle has quit IRC (Read error: Connection reset by peer)
02:11 🔗 jodizzle_ has joined #archiveteam-bs
02:13 🔗 phuzion has quit IRC (Read error: Operation timed out)
02:14 🔗 jodizzle_ is now known as jodizzle
02:42 🔗 lindalap has joined #archiveteam-bs
02:43 🔗 lindalap https://recording.tk.informatik.tu-darmstadt.de/
02:43 🔗 lindalap found randomly
02:43 🔗 lindalap not saved at Wayback Machine either
02:43 🔗 katocala has quit IRC (Ping timeout: 496 seconds)
02:44 🔗 katocala has joined #archiveteam-bs
02:52 🔗 Meli has joined #archiveteam-bs
03:08 🔗 qw3rty_ has joined #archiveteam-bs
03:15 🔗 qw3rty__ has quit IRC (Read error: Operation timed out)
03:39 🔗 katocala has quit IRC (Ping timeout: 272 seconds)
03:40 🔗 katocala has joined #archiveteam-bs
03:57 🔗 Ajay1 has quit IRC (Quit: Goodbye)
04:08 🔗 Yurume has quit IRC (Quit: No Ping reply in 180 seconds.)
04:09 🔗 Yurume has joined #archiveteam-bs
04:24 🔗 DogsRNice has quit IRC (Read error: Connection reset by peer)
04:25 🔗 lindalap has quit IRC (Quit: lindalap)
05:11 🔗 OrIdow6 has quit IRC (Remote host closed the connection)
05:12 🔗 OrIdow6 has joined #archiveteam-bs
06:33 🔗 kiska has quit IRC (Read error: Connection reset by peer)
06:35 🔗 kiska has joined #archiveteam-bs
06:39 🔗 atphoenix has quit IRC (Read error: Connection reset by peer)
06:40 🔗 atphoenix has joined #archiveteam-bs
07:44 🔗 britmob_ has joined #archiveteam-bs
07:46 🔗 benjinsmi has joined #archiveteam-bs
07:47 🔗 atphoenix has quit IRC (Ping timeout: 265 seconds)
07:47 🔗 britmob has quit IRC (Ping timeout: 265 seconds)
07:48 🔗 benjinss has quit IRC (Ping timeout: 265 seconds)
07:48 🔗 atphoenix has joined #archiveteam-bs
08:11 🔗 BlueMax has quit IRC (Quit: Leaving)
08:15 🔗 godane has quit IRC (Read error: Operation timed out)
08:17 🔗 godane has joined #archiveteam-bs
09:13 🔗 schbirid has joined #archiveteam-bs
09:13 🔗 katocala has quit IRC (Read error: Operation timed out)
09:13 🔗 katocala has joined #archiveteam-bs
09:27 🔗 brayden has quit IRC (Ping timeout: 272 seconds)
09:32 🔗 brayden has joined #archiveteam-bs
09:38 🔗 BlueMax has joined #archiveteam-bs
09:40 🔗 brayden has quit IRC (Ping timeout: 272 seconds)
09:45 🔗 brayden has joined #archiveteam-bs
09:51 🔗 Arcorann has quit IRC (Read error: Connection reset by peer)
10:00 🔗 Ryz has quit IRC (Quit: Ping timeout (120 seconds))
10:00 🔗 Ryz has joined #archiveteam-bs
10:20 🔗 schbirid is there a way to set a referrer when using http://web.archive.org/save
10:20 🔗 schbirid ?
10:30 🔗 Arcorann has joined #archiveteam-bs
10:51 🔗 arkiver no
11:14 🔗 brayden has quit IRC (Ping timeout: 272 seconds)
11:18 🔗 brayden has joined #archiveteam-bs
12:02 🔗 Jens has quit IRC (Remote host closed the connection)
12:03 🔗 Jens has joined #archiveteam-bs
12:09 🔗 BlueMax has quit IRC (Quit: Leaving)
12:17 🔗 atphoenix has quit IRC (Ping timeout: 265 seconds)
12:17 🔗 atphoenix has joined #archiveteam-bs
13:00 🔗 Arcorann I recently stumbled on a link to a 4chan archive https://yuki.la and noticed it wasn't mentioned at the wiki, probably worth looking into info for that site
13:03 🔗 HP_Archiv has joined #archiveteam-bs
13:38 🔗 HP_Archiv has quit IRC (Quit: Leaving)
13:58 🔗 acridAxid has quit IRC (Quit: marauder)
14:09 🔗 arkiver BartoCH: hi! did everything go well with setting up wget-at?
14:11 🔗 BartoCH git clone && docker build -t pastebin-grab . && docker run --detach --name pastebin-grab -v /home/barto/warrior/pastebin-data/:/grab/data/ pastebin-grab:latest Barto --concurrent=20
14:12 🔗 BartoCH that is all i had to do
14:12 🔗 BartoCH ofc i had an impromptu git pull to do ;)
14:13 🔗 arkiver ah yes
14:13 🔗 kiska Come to hackint?
14:13 🔗 arkiver well good!
14:13 🔗 kiska Also are your machines getting connection limit reached?
14:13 🔗 kiska I may have to increase the alloc
14:14 🔗 BartoCH i have so403 me retfinished stuffs yeah
14:14 🔗 BartoCH damn, internet is laggy now :D
14:14 🔗 kiska No rsync issues :D
14:14 🔗 arkiver oof me too
14:14 🔗 BartoCH aaand, i was banned by pastebin
14:14 🔗 kiska 403's or conn -1 or conn 100?
14:14 🔗 BartoCH Pastebin.com has blocked your IP
14:14 🔗 arkiver yep same
14:14 🔗 BartoCH 403
14:15 🔗 kiska Ah so not my target :D
14:15 🔗 arkiver kiska: no
14:15 🔗 BartoCH arkiver: we kind of got stuck at the same 300 items point
14:15 🔗 kiska https://server8.kiska.pw/uploads/feb324249491be79/image.png
14:15 🔗 kiska Circled is NVMe drives flushing cache :D
14:16 🔗 kiska Time for some hetzner? :P
14:16 🔗 kiska Does the 403 tell us how long they ban for?
14:17 🔗 arkiver no
14:17 🔗 arkiver but it's temporary
14:18 🔗 kiska Well thats good at least :D
14:29 🔗 Meli has quit IRC (Remote host closed the connection)
14:30 🔗 Meli has joined #archiveteam-bs
14:58 🔗 Meli has quit IRC (Remote host closed the connection)
14:58 🔗 Meli has joined #archiveteam-bs
15:01 🔗 Meli has quit IRC (Remote host closed the connection)
15:02 🔗 Meli has joined #archiveteam-bs
15:19 🔗 godane has quit IRC (Read error: Connection reset by peer)
15:21 🔗 Meli has quit IRC (Remote host closed the connection)
15:23 🔗 brayden has quit IRC (Ping timeout: 272 seconds)
15:26 🔗 Meli has joined #archiveteam-bs
15:27 🔗 brayden has joined #archiveteam-bs
15:59 🔗 Arcorann has quit IRC (Read error: Connection reset by peer)
16:08 🔗 DogsRNice has joined #archiveteam-bs
16:14 🔗 DogsRNice has quit IRC (Ping timeout: 265 seconds)
16:16 🔗 Gfy_ is now known as Gfy
16:18 🔗 DogsRNice has joined #archiveteam-bs
17:12 🔗 godane has joined #archiveteam-bs
17:41 🔗 wyatt8740 has quit IRC (Read error: Operation timed out)
17:42 🔗 Ryz has quit IRC (Remote host closed the connection)
17:42 🔗 kiska1825 has quit IRC (Remote host closed the connection)
17:42 🔗 Ryz has joined #archiveteam-bs
17:42 🔗 kiska1825 has joined #archiveteam-bs
17:43 🔗 wyatt8740 has joined #archiveteam-bs
17:53 🔗 Meli has quit IRC (Remote host closed the connection)
17:55 🔗 Meli has joined #archiveteam-bs
18:50 🔗 exoire has joined #archiveteam-bs
18:55 🔗 wyatt8740 has quit IRC (Read error: Operation timed out)
18:57 🔗 wyatt8740 has joined #archiveteam-bs
19:16 🔗 zyphlar_ has joined #archiveteam-bs
19:58 🔗 Stiletto has quit IRC (Read error: Operation timed out)
20:00 🔗 Stiletto has joined #archiveteam-bs
20:07 🔗 asdf0101 has quit IRC (The Lounge - https://thelounge.chat)
20:08 🔗 asdf0101 has joined #archiveteam-bs
20:09 🔗 exoire has quit IRC (Client terminated!)
20:22 🔗 asdf0101 has quit IRC (Remote host closed the connection)
20:25 🔗 asdf0101 has joined #archiveteam-bs
20:27 🔗 Dj-Wawa_ has quit IRC (Quit: Dj-Wawa_)
20:28 🔗 Dj-Wawa has joined #archiveteam-bs
20:37 🔗 Dj-Wawa has quit IRC (Quit: Dj-Wawa)
20:38 🔗 Dj-Wawa has joined #archiveteam-bs
20:41 🔗 godane SketchCow: looks like manualsbase.com has less dups at 11k ids
20:42 🔗 godane first 16 of that range had zero file sizes that was the same
20:44 🔗 godane nevermind
20:45 🔗 godane either way every id is getting uploaded cause metadata title could be better
21:26 🔗 zyphlar_ has quit IRC (Quit: Connection closed for inactivity)
21:28 🔗 katocala has quit IRC (Ping timeout: 745 seconds)
21:28 🔗 katocala has joined #archiveteam-bs
21:35 🔗 katocala has quit IRC (Ping timeout: 255 seconds)
21:35 🔗 katocala has joined #archiveteam-bs
22:14 🔗 acridAxid has joined #archiveteam-bs
22:45 🔗 britmob_ has quit IRC (Ping timeout: 265 seconds)
22:47 🔗 britmob has joined #archiveteam-bs
23:47 🔗 Stiletto has quit IRC (Read error: Connection reset by peer)
23:49 🔗 Stiletto has joined #archiveteam-bs
23:52 🔗 Stiletto has quit IRC (Read error: Connection reset by peer)
23:52 🔗 Stiletto has joined #archiveteam-bs
23:54 🔗 wyatt8740 has quit IRC (Read error: Operation timed out)
23:55 🔗 wyatt8740 has joined #archiveteam-bs
23:57 🔗 Mateon1 has quit IRC (Read error: Operation timed out)
23:57 🔗 Mateon1 has joined #archiveteam-bs

irclogger-viewer