[00:11] *** katocala has quit IRC (Ping timeout: 255 seconds) [00:11] *** katocala has joined #archiveteam-bs [00:26] *** BlueMax has joined #archiveteam-bs [00:31] *** Arcorann has joined #archiveteam-bs [00:44] We now have a new customized wget with full ZSTD compression support [00:45] with ZSTD we can use trained custom dictionaries, which overall decrease the size of certain WARC records enormously [00:45] a trainer is in place which will train a dictionary on the latest WARC record uploaded to IA for a certain project. [00:46] the new dictionary is then used by warrior and workers to compress data with [00:47] this can decrease the size of data with tens of percentages overall, which means less outgoing bandwidth, and less disk usage on IA [00:57] Does this work in wget-lua, or just plain wget? [00:58] it's wget-at, currently at https://github.com/ArchiveTeam/wget-lua/tree/v1.20.3-at and will be merged with master later [00:59] it's renamed to wget-at (from wget-lua) since this now includes more than just the Lua scripting option [00:59] it also includes the deduplication, URL agnostic deduplication option, and ZSTD compression with dictionaries [01:10] *** DLoader_ has joined #archiveteam-bs [01:20] *** DLoader has quit IRC (Ping timeout: 745 seconds) [01:20] *** DLoader_ is now known as DLoader [01:36] *** lennier2 has joined #archiveteam-bs [01:48] *** lennier1 has quit IRC (Ping timeout: 745 seconds) [01:48] *** lennier2 is now known as lennier1 [01:53] *** lennier1 has quit IRC (Quit: Going offline, see ya! (www.adiirc.com)) [01:54] *** Meli has quit IRC (Read error: Operation timed out) [01:55] *** Frogging has quit IRC (Read error: Operation timed out) [01:55] *** Frogging has joined #archiveteam-bs [01:55] *** legoktm has quit IRC (Read error: Operation timed out) [01:56] *** legoktm has joined #archiveteam-bs [01:56] *** Ctrl has quit IRC (Read error: Operation timed out) [01:57] *** Raccoon` has joined #archiveteam-bs [01:58] *** wessel152 has quit IRC (Read error: Operation timed out) [02:00] *** Raccoon has quit IRC (Read error: Operation timed out) [02:00] *** Raccoon` is now known as Raccoon [02:03] *** Pixi has quit IRC (Read error: Connection reset by peer) [02:03] *** Xibalba has quit IRC (Read error: Operation timed out) [02:03] *** Xibalba has joined #archiveteam-bs [02:03] *** Pixi has joined #archiveteam-bs [02:03] *** Xibalba has quit IRC (Handshake flooding) [02:03] *** Xibalba has joined #archiveteam-bs [02:04] *** lennier1 has joined #archiveteam-bs [02:04] *** phuzion_ has joined #archiveteam-bs [02:11] *** jodizzle has quit IRC (Read error: Connection reset by peer) [02:11] *** jodizzle_ has joined #archiveteam-bs [02:13] *** phuzion has quit IRC (Read error: Operation timed out) [02:14] *** jodizzle_ is now known as jodizzle [02:42] *** lindalap has joined #archiveteam-bs [02:43] https://recording.tk.informatik.tu-darmstadt.de/ [02:43] found randomly [02:43] not saved at Wayback Machine either [02:43] *** katocala has quit IRC (Ping timeout: 496 seconds) [02:44] *** katocala has joined #archiveteam-bs [02:52] *** Meli has joined #archiveteam-bs [03:08] *** qw3rty_ has joined #archiveteam-bs [03:15] *** qw3rty__ has quit IRC (Read error: Operation timed out) [03:39] *** katocala has quit IRC (Ping timeout: 272 seconds) [03:40] *** katocala has joined #archiveteam-bs [03:57] *** Ajay1 has quit IRC (Quit: Goodbye) [04:08] *** Yurume has quit IRC (Quit: No Ping reply in 180 seconds.) [04:09] *** Yurume has joined #archiveteam-bs [04:24] *** DogsRNice has quit IRC (Read error: Connection reset by peer) [04:25] *** lindalap has quit IRC (Quit: lindalap) [05:11] *** OrIdow6 has quit IRC (Remote host closed the connection) [05:12] *** OrIdow6 has joined #archiveteam-bs [06:33] *** kiska has quit IRC (Read error: Connection reset by peer) [06:35] *** kiska has joined #archiveteam-bs [06:39] *** atphoenix has quit IRC (Read error: Connection reset by peer) [06:40] *** atphoenix has joined #archiveteam-bs [07:44] *** britmob_ has joined #archiveteam-bs [07:46] *** benjinsmi has joined #archiveteam-bs [07:47] *** atphoenix has quit IRC (Ping timeout: 265 seconds) [07:47] *** britmob has quit IRC (Ping timeout: 265 seconds) [07:48] *** benjinss has quit IRC (Ping timeout: 265 seconds) [07:48] *** atphoenix has joined #archiveteam-bs [08:11] *** BlueMax has quit IRC (Quit: Leaving) [08:15] *** godane has quit IRC (Read error: Operation timed out) [08:17] *** godane has joined #archiveteam-bs [09:13] *** schbirid has joined #archiveteam-bs [09:13] *** katocala has quit IRC (Read error: Operation timed out) [09:13] *** katocala has joined #archiveteam-bs [09:27] *** brayden has quit IRC (Ping timeout: 272 seconds) [09:32] *** brayden has joined #archiveteam-bs [09:38] *** BlueMax has joined #archiveteam-bs [09:40] *** brayden has quit IRC (Ping timeout: 272 seconds) [09:45] *** brayden has joined #archiveteam-bs [09:51] *** Arcorann has quit IRC (Read error: Connection reset by peer) [10:00] *** Ryz has quit IRC (Quit: Ping timeout (120 seconds)) [10:00] *** Ryz has joined #archiveteam-bs [10:20] is there a way to set a referrer when using http://web.archive.org/save [10:20] ? [10:30] *** Arcorann has joined #archiveteam-bs [10:51] no [11:14] *** brayden has quit IRC (Ping timeout: 272 seconds) [11:18] *** brayden has joined #archiveteam-bs [12:02] *** Jens has quit IRC (Remote host closed the connection) [12:03] *** Jens has joined #archiveteam-bs [12:09] *** BlueMax has quit IRC (Quit: Leaving) [12:17] *** atphoenix has quit IRC (Ping timeout: 265 seconds) [12:17] *** atphoenix has joined #archiveteam-bs [13:00] I recently stumbled on a link to a 4chan archive https://yuki.la and noticed it wasn't mentioned at the wiki, probably worth looking into info for that site [13:03] *** HP_Archiv has joined #archiveteam-bs [13:38] *** HP_Archiv has quit IRC (Quit: Leaving) [13:58] *** acridAxid has quit IRC (Quit: marauder) [14:09] BartoCH: hi! did everything go well with setting up wget-at? [14:11] git clone && docker build -t pastebin-grab . && docker run --detach --name pastebin-grab -v /home/barto/warrior/pastebin-data/:/grab/data/ pastebin-grab:latest Barto --concurrent=20 [14:12] that is all i had to do [14:12] ofc i had an impromptu git pull to do ;) [14:13] ah yes [14:13] Come to hackint? [14:13] well good! [14:13] Also are your machines getting connection limit reached? [14:13] I may have to increase the alloc [14:14] i have so403 me retfinished stuffs yeah [14:14] damn, internet is laggy now :D [14:14] No rsync issues :D [14:14] oof me too [14:14] aaand, i was banned by pastebin [14:14] 403's or conn -1 or conn 100? [14:14] Pastebin.com has blocked your IP [14:14] yep same [14:14] 403 [14:15] Ah so not my target :D [14:15] kiska: no [14:15] arkiver: we kind of got stuck at the same 300 items point [14:15] https://server8.kiska.pw/uploads/feb324249491be79/image.png [14:15] Circled is NVMe drives flushing cache :D [14:16] Time for some hetzner? :P [14:16] Does the 403 tell us how long they ban for? [14:17] no [14:17] but it's temporary [14:18] Well thats good at least :D [14:29] *** Meli has quit IRC (Remote host closed the connection) [14:30] *** Meli has joined #archiveteam-bs [14:58] *** Meli has quit IRC (Remote host closed the connection) [14:58] *** Meli has joined #archiveteam-bs [15:01] *** Meli has quit IRC (Remote host closed the connection) [15:02] *** Meli has joined #archiveteam-bs [15:19] *** godane has quit IRC (Read error: Connection reset by peer) [15:21] *** Meli has quit IRC (Remote host closed the connection) [15:23] *** brayden has quit IRC (Ping timeout: 272 seconds) [15:26] *** Meli has joined #archiveteam-bs [15:27] *** brayden has joined #archiveteam-bs [15:59] *** Arcorann has quit IRC (Read error: Connection reset by peer) [16:08] *** DogsRNice has joined #archiveteam-bs [16:14] *** DogsRNice has quit IRC (Ping timeout: 265 seconds) [16:16] *** Gfy_ is now known as Gfy [16:18] *** DogsRNice has joined #archiveteam-bs [17:12] *** godane has joined #archiveteam-bs [17:41] *** wyatt8740 has quit IRC (Read error: Operation timed out) [17:42] *** Ryz has quit IRC (Remote host closed the connection) [17:42] *** kiska1825 has quit IRC (Remote host closed the connection) [17:42] *** Ryz has joined #archiveteam-bs [17:42] *** kiska1825 has joined #archiveteam-bs [17:43] *** wyatt8740 has joined #archiveteam-bs [17:53] *** Meli has quit IRC (Remote host closed the connection) [17:55] *** Meli has joined #archiveteam-bs [18:50] *** exoire has joined #archiveteam-bs [18:55] *** wyatt8740 has quit IRC (Read error: Operation timed out) [18:57] *** wyatt8740 has joined #archiveteam-bs [19:16] *** zyphlar_ has joined #archiveteam-bs [19:58] *** Stiletto has quit IRC (Read error: Operation timed out) [20:00] *** Stiletto has joined #archiveteam-bs [20:07] *** asdf0101 has quit IRC (The Lounge - https://thelounge.chat) [20:08] *** asdf0101 has joined #archiveteam-bs [20:09] *** exoire has quit IRC (Client terminated!) [20:22] *** asdf0101 has quit IRC (Remote host closed the connection) [20:25] *** asdf0101 has joined #archiveteam-bs [20:27] *** Dj-Wawa_ has quit IRC (Quit: Dj-Wawa_) [20:28] *** Dj-Wawa has joined #archiveteam-bs [20:37] *** Dj-Wawa has quit IRC (Quit: Dj-Wawa) [20:38] *** Dj-Wawa has joined #archiveteam-bs [20:41] SketchCow: looks like manualsbase.com has less dups at 11k ids [20:42] first 16 of that range had zero file sizes that was the same [20:44] nevermind [20:45] either way every id is getting uploaded cause metadata title could be better [21:26] *** zyphlar_ has quit IRC (Quit: Connection closed for inactivity) [21:28] *** katocala has quit IRC (Ping timeout: 745 seconds) [21:28] *** katocala has joined #archiveteam-bs [21:35] *** katocala has quit IRC (Ping timeout: 255 seconds) [21:35] *** katocala has joined #archiveteam-bs [22:14] *** acridAxid has joined #archiveteam-bs [22:45] *** britmob_ has quit IRC (Ping timeout: 265 seconds) [22:47] *** britmob has joined #archiveteam-bs [23:47] *** Stiletto has quit IRC (Read error: Connection reset by peer) [23:49] *** Stiletto has joined #archiveteam-bs [23:52] *** Stiletto has quit IRC (Read error: Connection reset by peer) [23:52] *** Stiletto has joined #archiveteam-bs [23:54] *** wyatt8740 has quit IRC (Read error: Operation timed out) [23:55] *** wyatt8740 has joined #archiveteam-bs [23:57] *** Mateon1 has quit IRC (Read error: Operation timed out) [23:57] *** Mateon1 has joined #archiveteam-bs