Giganews header compression posted Tue, 25 May 2010 02:22:38 UTC

permalink

After messing around a bit with TCL, I finally figured out how to read the compressed headers from Giganews. Yay.

Thanks to a post over here, I was able to start with the basic NNTP conversation and add the rest I pieced together over the past couple of nights. My version with compression and SSL looks like this:

#!/usr/bin/tclsh

# load tls package
package require tls

# configure socket
set sock [tls::socket news.giganews.com 563]
fconfigure $sock -encoding binary -translation crlf -buffering none

# authenticate to GN
puts stderr [gets $sock]
puts stderr "sending user name"
puts $sock "authinfo user xxxxxxxx"
puts stderr [gets $sock]
puts stderr "sending password"
puts $sock "authinfo pass yyyyyyyy"
puts stderr [gets $sock]

# enable compression
puts stderr "sending xfeature command"
puts $sock "xfeature compress gzip"
puts stderr [gets $sock]

# set group
puts stderr "sending group command"
puts $sock "group giganews.announce"
puts stderr [gets $sock]

# issue xover command based on group posts
puts stderr "sending xover command"
puts $sock "xover 2-48"
set resp [gets $sock]
puts stderr $resp

# if the response is 224, parse the results
if {[lindex [split $resp] 0] == "224"} {

# loop through uncompressed results
#       while {[gets $sock resp] > 0} {
#               if {$resp == "."} {
#                       puts stdout $resp
#                       break
#               }
#               puts stdout $resp
#       }

# loop through compressed results
        while {[gets $sock resp] > 0} {
                if {[string index $resp end] == "."} {
                        append buf [string range $resp 0 end-1]
                        break
                }
                append buf $resp
        }
}

# uncompress those headers!
puts -nonewline stdout [zlib decompress $buf]

# issue a quit command
puts stderr "sending quit"
puts $sock quit
puts -nonewline stderr [read $sock]

Feel free to take the results and run. I'm not sure if there is a limit to how many headers you can fetch in a single go. I imagine it's more or less limited to your local buffer size, so don't grab too many at a time (at least in TCL I imagine). Anything more aggressive would require some fine tuning no doubt. But this was all just proof of concept to see if I could make it work. Now to write my newzbin.com replacement!