[04:40] *** tech234a has joined #webroasting [06:50] *** tech234a has quit IRC (Quit: Connection closed for inactivity) [15:03] *** tech234a has joined #webroasting [17:23] *** tech234a has quit IRC (Quit: Connection closed for inactivity) [18:25] *** kiska is now known as kiskan [18:29] *** kiskan is now known as kiskablah [18:30] *** kiskablah is now known as kiska3 [19:29] *** kiska3 is now known as kiska [20:58] OK, the custom search stuff looked more trouble than it's worth, but I was able to get a couple of hundred URLs out pretty quickly by using https://serpapi.com/demo [21:19] Bing's results are ridiculously bad, and there is a lot of duplication in there. [21:19] wow, you're not kidding. [22:36] when running grab-site and it stops, but doesn't seem to terminate, does that still mean it's done? [22:38] eythian: My theory is that Bing only tolerates bots because nobody else wants to use it due to how bad the results are. ;-) [22:39] that seems plausible :) [22:39] That way they can boost their usage numbers. [22:40] "stops but doesn't terminate" sounds like the wpull bug we've been seeing a lot on ArchiveBot pipelines. But I don't know since I've never used grab-site. You might want to ask ivan over in -ot. [22:40] ta will do