Tag Archives: binaries

What is a newsgroup

I have seen a lot of discussions on forums and even on Usenet itself starting with questions like “what is a news group”, “what are newsgroups” or more commonly “what is a newsgroup”.  A Usenet newsgroup is a discussion group for a specific topic, which is shared on Usenet.  It is not a forum, or a blog, or even a file-sharing or online storage application, although it can look like any of those depending on what newsgroup you are reading. There are big differences between Usenet and other types of discussion sites and file-sharing applications.

A forum is typically hosted on a single web server, in order to post to that forum you typically have to sign up for an account to post questions or answers. Usenet is hosted on literally thousands of newsservers around the world, each server has it’s own copy of articles that it carries.  Most servers require users to sign up for an account, but there are some with free accounts and a few that still allow anonymous connections.  If a persons Usenet account allows posting, they can post questions and answers to a newsgroup and it is copied to all the servers in the network.  This gives excellent redundancy, which is a definite benefit, but there is a lack of control that most forums allow, such as being able to block some discussions or delete certain comments.

A blog is also typically hosted on a single web server using software like wordpress. Usually the blog owner is the only person allowed to post, although many sites allow others to leave comments. Bloggers typically post discussions about topics they are interested in, such as how their day went or what their cat is up too, or in this case “what are news groups”. A person could do exactly the same thing in a Usenet newsgroup, with all the same Usenet benefits and concerns listed above. One big difference between a newsgroup and a blog is that once an article is posted, you cannot edit it, and sometimes getting it deleted from all those servers is difficult, many servers ignore cancel requests for various reasons, including prevent vandalism.

File-sharing applications allow people to host binaries on their personal computers and share them with other people over the internet using a p2p (peer-to-peer) connection so users can copy files to and from each others machines.  Although Usenet was not originally designed to share  binaries, it was not long after its inception that people figured out how to use it for that.  There are a couple of glaring differences between Usenet and P2P, the biggest being the distributed way it stores files.  Although P2P usually has good distribution of files, if everyone with the binary you are interested in has deleted it from their PC or are simply not running the application, you cannot download the binary.  Another issue can be if the person that has parts of the file you want has a slow connection, it can take a very long time to download it. Getting a binary from a Usenet servers is easier and usually faster, if the binary is listed you can download it all from your newsserver, and the download speed is constant.  Most modern Usenet providers have very fast internet connections and can feed binaries to multiple users at the same time without any significant loss of download speed.

Online storage applications allow a user to securely save a backup copy of files to the providers server.  The provider usually has redundant servers and performs backups to protect the users data so they can recover at any time in the future.  They also usually have applications for many different devices including PCs, Macs and Smart phones. Online storage is really geared towards allowing a person or a company to backup their files somewhere else so they won’t lose them in the case of a fire or other catastrophe. Usenet allows users to post files, and due to its distributed nature it seems like a fairly safe way to store your personal information, but there is no inherent security; a file posted to a newsgroup can be viewed by anyone with a newsreader.  Users can encrypt their postings to prevent prying eyes from looking at personal information, but some servers may delete your files automatically based on a normal expiration schedule.  Because of the lack of security and guarantee of data retention, Usenet cannot be relied on as a file storage medium for critical data for an extended period of time.

Google Group vs Usenet Newsgroup

Earlier today I got an email from a user that was wanting to know the difference between googlegroups and Usenet newsgroups.  They went on further to ask which was a better choice for them based on some criteria.  The were looking for images, so my immediate response for them was “go with a Usenet server”, but then I realized the reasons were as important as my answer.

Short History

Usenet newsgroups have been around since the late 70s/early 80s.  I have operated a server since just before the “great renaming” back in 1987.  My personal experience is that the Usenet has been a wonderful tool for sharing of all kinds from the beginning.

Close to the beginning some people got the bright idea to keep copies of all the articles posted to Usenet.  The Deja News archive of Usenet articles, allowed visitors to their website to perform extensive searches and people could sign up for a free account to post articles in text newsgroups.  It was a wonderful tool, until they tried to make money from it.  In about 1999 they tried to monetize it, and everything went sideways.

Googlegroups

On Feb 12, 2001 Google bought the archives and other resources that were left to the, by then failed, Deja News.  Google used those archives to start Googlegroups.  Google groups are fundamentally the same as Usenet newsgroups, and many of the groups are still peered with Tier-1 providers on the internet.  So, users that post to a Google group about ‘dog behavior’ might get a response from someone using a Usenet server reading ‘rec.pets.dogs.behavior’, and vice versa.

Probably due to the governing board in charge of the “Big 8” hierarchies on Usenet, Google groups created by their users are not created on the Usenet.  Because of this, not all groups will get peered with Usenet servers.  This limits the amount of low-bandwidth or nonsense groups in Usenet, but also lessens the possible amount of content it could provide.

Okay… which is better

Neither, both, either… really depends on your needs.  All things being equal, either should work for someone that is just looking for a discussion group about a topic.  There are many considerations that can change that though.

Considerations
Google Group Usenet Newsgroup
Create Group Y Y1
Subscription N Y2
Software Required N Y/N3
Mailing List4 Y N
Binaries (alt.binaries.*) N Y
Document Sharing5 Y Y

1 – It is possible to create a newsgroup in the alt.* hierarchy just by typing its name into the newsgroup field of your newsgroup client.  If you want to create a newsgroup in the “Big 8” hierarchies you have to prove there is need/interest through a vetting process.  Few new newsgroups are created any more in the “Big 8”.

2 – It is possible to get a free account from some Usenet providers, these are usually speed or size limited and do not have binaries, but if your goal is to join a discussion group, that would be sufficient

3 – there are a several options for reading the Usenet newsgroups

  1. You can purchase a newsreader for Usenet, typical cost is around $20 and some require a subscription for advanced search features.
  2. Some Usenet providers (such as EasyNews) have a web-based interface which is actually much more sophisticated than the one provided by Googlegroups, they support searching for and previewing binaries
  3. there are free newsreaders, but they are normally designed more for Usenet enthusiasts than for casual users

4 – there is no mailing list on Usenet unless you create one… which is very angrily frowned upon.  A posting to a newsgroup is visible to all subscribers; you probably have a better chance of someone reading a posting in a newsgroup than getting passed their spam filter for mailing them directly

5 – document sharing in Google groups is limited.  Although document sharing is not directly supported on Usenet, you can post binaries in some groups.  You could post a document in an alt.binaries.* group, then reference it in an article in a discussion newsgroup.  There is a program (called uBackup) which can be used to “store” documents on the Usenet.

What is a Newsgroup

A Usenet newsgroup is a discussion group within the Usenet system for messages posted by users about a specific topic.  This term is often confused with forums or blogs, but they are different in many ways. In order to read newsgroups you need access to a Usenet server and newsreader software.

Usenet was originally based on the old bulletin board systems (BBS) of the late 70’s and early 80’s.  The primary difference is that a BBS required users to dial in to a centralized computer to read all the articles, and they were not (typically) shared between different BBS’.  Usenet turned that design on it’s head, articles are stored and copied between servers around the world, with each maintaining a separate repository of articles for all the newsgroups.  When servers communicated with each other they would trade missing articles so that all servers eventually would have all articles in their repository.  This design allowed users to dial in to a newsgroup server and share ideas with people from all around the world.

Newsgroup Hierarchies

Newsgroups are grouped together in hierarchies which help users to locate newsgroups they are interested in more easily on the Usenet.  Originally all newsgroups were in the net.* hierarchy, and locating a news group you were interested in could be difficult. In 1987 some of the major Usenet backbone sites got together and rearranged all the newsgroups in what was called the “big-7”, they were:

  • comp.* – computer related newsgroups
  • news.* – Usenet related newsgroups
  • rec.* – recreational activity related newsgroups
  • sci.* – newsgroups for scientific topics
  • soc.* – social issue newsgroups
  • talk.* – contentions issue related newsgroups suck as religion and politics
  • misc.* – miscellaneous discussions that didn’t fit any of the above

In 1995 the “Big 7” was changed to the “Big 8” with the addition of the humanities.* hierarchy for discussion topics such as the arts and philosophy. All of the “Big 8” groups were controlled by a group from the major backbone operators and they allowed users to make suggestions for creating or deleting groups, and those requests were formalized by choosing an appropriate name and group charter.  Finally, the proposal was put to a vote by the group.  Often, in the end, the answer was “No” which frustrated a lot of users who pushed for the creation of a new hierarchy that was not controlled by the “Big 8” group and eventually the alt.* hierarchy was created.

Unlike the “Big 8” which requires users to present a Request for Discussion (RFD) to create a newsgroup, the alt.* hierarchy is the wild west.  Users that want to create their own newsgroup simply put alt.<topic> in the newsgroup to post to and many servers will propagate it to all their peers.  The major shortfall in this design is that; while “Big 8” newsgroups will be carried by almost all Usenet servers, the decision to pass on unknown groups is up to the individual server operators, and some will refuse to carry or propagate those groups.  The upshot of this is that you can create a group, but no one may ever see it but you.

Other Hierarchies

There are many other hierarchies now on the Usenet, many of which are operator or country/language based.  Because of the flexible design of the Usenet software, an operator can create newsgroups which are only available to their users and will not be passed to other servers.  Also, for many non-English users, groups have been created that allow discussion in their country or native language.  Most of these hierarchies have a regulating body (or individual) which maintains the list of allowed newsgroups.  Some of the major hierarchies include:

  • cn.* – for Chinese newsgroups
  • de.* – for German newsgroups
  • fr.* – for French newsgroups
  • it.* – for Italian newsgroups
  • kr.* – for Korean newsgroups
  • tw.* – for Taiwanese newsgroups

Types of Newsgroups

There are predominantly two types of newsgroups, binary or text.  Both work the same way from a server standpoint, and there is no major differences in the naming conventions except that binary groups often have ‘binaries’ somewhere in the name, such as ‘alt.binaries.pictures.rail’ which carries pictures of trains by newsgroup readers.  Often there is a discussion newsgroup for a binary group which will have an identical name but have ‘.d’ added to the end denoting that it is for discussion of postings to the binary newsgroup.

Newsgroups are usually about specific topics, and some are even moderated to ensure that off-topic posts are not posted to a discussion, while others allow users to post articles about a wide variety of topics.  There are currently over 110,000 newsgroups, most of which do not see any activity except for SPAM, but estimates are that about 20,000 of the newsgroups are still regularly used by people for their intended purposes.  A lot of the other 90,000+ are newsgroups created in the alt.* hierarchy either intentionally, or often through a typographical error while typing in the newsgroup to post or cross-post to.

Binary Newsgroups

The Usenet was originally designed to work with US-ASCII characters, and was at that time a very US-centric system.  The Usenet originally only connected a few Universities together, and that was expanded to include corporations, high-schools and some individuals.  It wasn’t long before some users wanted to be able to send binary data in newsgroups and a method (called UUencode) was created which converted 8-bit binary files to 7-bit US-ASCII.  Users that wanted to use the binary data could decode it (using UUdecode) and have a copy of the original file.

The modern Usenet software is designed to work with many international character sets and can transfer those articles without risk of damage to 8-bit data, but articles are still often sent in 7-bit clean formats.  There are also newer programs that can create articles that use 8-bit data such as yEnc.  yEnc’s 8-bit articles are much more efficient at transferring binaries than the older UUencode articles because the conversion from 8-bit to 7-bit has an approximate overhead of about 40%, whereas the overhead for yEnc is much lower, usually as low as 1-2%.

Another restriction of the original Usenet servers was often article size, this restriction required that larger binaries be broken up into multiple parts which they poster would then post manually.  Users that wanted the original binary would download all the parts, and put them back together manually before decoding. This process was tedious and error prone, plus servers often would not get all the parts of a posting, or might delete parts before the user had time to download all of them.  The high error rate caused the ‘.d’ discussion groups to often be filled with requests for “part 13 of 53”.  Modern newsgroup readers are able to handle multi-part binary postings internally and the users usually needs to just pick the binary they want and come back later for the completed download.

Downloading Usenet Binaries

Many years ago, the Usenet only allowed 7-bit text messages to ensure that communications was possible between any type of machine, using any kind of modem.  It was not long before someone figured out how to encode 8-bit data into a 7-bit format so that binaries could be sent using Usenet with no modifications to the Newsgroup server software.  The original programs for encoding and decoding articles were named uuencode and uudecode, respectively.  Modern versions of Usenet are now 8-bit safe and even allow for multi-byte text in UTF-8 format, but various encoding methods for articles are still used.

In the old days, to download a Usenet binary, you had to locate all its parts on the server and download them to your computer.  After all the parts were on your computer you would feed them to a decoding program, which usually gave you the binary or an archive containing your binary.  If your binary was in an archive, you then used another application to extract your binary from the archive.  Needless to say, this could be a long and tedious process, and often not all the parts were available on your Usenet server.

Over the years, the applications and Newsgroup servers have gotten much more sophisticated.

  • Usenet server software was upgraded to be able to work with other languages than English, and in the process was made 8-bit safe so that encoding from 8-bit to 7-bit was no longer required, decreasing article sizes by about 40%
  • Web sites such as Newzbin started popping up which created NZB files providing users with an exact list of all articles required to recreate a binary
  • Encoding software began creating NZB and SFV files automatically and posting them with the binaries
  • An application called parchive created ‘PAR’ files which could be used to repair incomplete files using complex algorithms that could not only detect errors in the parts, but could even recreate missing parts using the ‘PAR’ file and the parts that were successfully downloaded.

Most modern newsreaders have integrated newsgroup search features, and almost all of them support the NZB standard file format, both of which make finding all the parts of a Usenet binary much simpler.  In addition, most newsgroup readers also directly support decoding, repairing and extracting binaries into a directory for you, so all you have to do is tell it what to download with an NZB file and where to store it.  Another nice feature of some newsgroup browsers is that they can work with multiple servers, so if a part is missing from one server, they will attempt to get it from an alternate server.

Putting it all together

For most users interested in downloading from Usenet, the steps are now pretty simple.

  1. Choose a Usenet provider
  2. Choose a newsreader
  3. Use a newsgroup search website or your newsreader’s integrated search to find the digital media you want
  4. if you are using a website, import the NZB from the website
  5. click download
  6. when the download completes, use your new binary

Free Usenet Binaries

The truth about free Usenet

After you have spent some time trying the various free servers on my newsgroup search site, it should be apparent that it is no longer possible to download from the alt.binaries newsgroups for free. Many of us thought we had it all nailed down 20 years ago and were set; now we are going to the commercial Newsgroup providers because our university or corporate provider have stopped giving us access for free, due to increased cost and in some cases, legal concerns. Continue reading Free Usenet Binaries