An overview of Usenet
Long before Reddit and BitTorrent, there was Usenet. Think of it as the original online social network. Established in the early 1980s, Usenet was a cross between email and web forums used to exchange text files between users.
Fast forward to the 1990s, Usenet gets an upgrade to upload and share large binary files, such as videos, music, and images. But when BBS forums like Reddit and the BitTorrent protocol came around, Usenet was largely left behind.
This 36-year-old technology is far from dead, though. In fact, many consider it superior to torrenting for sharing and downloading large files, including copyrighted content and commercial software. So why haven’t you heard much about it? Because the first rule of Usenet is don’t talk about Usenet. Usenet providers and users don’t want to draw undue attention to themselves, lest it suffer the same fate as file sharing services like Napster, Limewire, and ThePirateBay. But the entertainment industry is now well aware of Usenet, so pretending we are protecting it by keeping our lips sealed would serve little purpose.
- 1 An overview of Usenet
- 2 What is Usenet?
- 3 Usenet vs. Torrents
- 4 Legal risks
- 5 Factors to consider when choosing a Usenet provider
- 6 What you need to start using Usenet
- 7 The history of Usenet
What is Usenet?
Usenet is a global distributed discussion system, although these days it’s more about file sharing than sending messages. It contains two types of files: text and binary. If you’re reading this, you’re probably more interested in the latter. Binary files include movies, music, software, games, and other large files. These files are indexed in a .NZB format, which allow the user to download all the pieces of a movie or game from various servers.
Users subscribe to a Usenet provider that hosts files for a monthly or yearly fee. The subscriber can search for an NZB file, similar to a torrent file, that allows him or her to download the content to a personal computer through a “newsreader” app.
Usenet vs. Torrents
Why would you pay for Usenet when you can just torrent? There are several reasons. First of all, Usenet downloads are much, much faster, because they come from centralized servers instead of other people’s computers. Instead of peers, Usenet can connect to dozens of servers at a time, which means you can download at speeds as fast as your ISP can handle.
Secondly, Usenet is private. The connections take place between you and the provider’s servers, and most providers offer an SSL-encrypted connection. Some even throw in VPNs for good measure. Torrents, on the other hand, require that you share at least some identifying information in order to connect to the tracker and peers.
Thirdly, downloading a Usenet file doesn’t mean you have to seed it for other users afterwards. Legally, this makes you less of a target because you’re not supplying other users with copyrighted content, and it consumes fewer computer resources and internet bandwidth.
Usenet providers make files available for a certain number of days. How many depends on the provider, but the standard is more than six years after the original posting. Until that time is up, users have full access to that file. Torrents only stay up as long as there are people seeding the file.
Usenet evangelists, if they were prone to talking about it, would probably tell you that no one has ever been arrested as a result of a Usenet download. That may be true, but there’s a first time for everything, so we recommend you use a VPN or at minimum download through an encrypted SSL connection. No arrests on record doesn’t mean the Usenet sector hasn’t come under scrutiny. Major providers and indexers (see: search engines) have been shut down through lawsuits in the past. And just because individual users haven’t gone to court doesn’t mean they weren’t targeted by law enforcement or angry lawyers.
Comparitech does not condone or encourage any violation of copyright restrictions. Please consider the law, victims, and risks of piracy before downloading copyrighted material without permission.
Factors to consider when choosing a Usenet provider
Retention period, transfer limits, server connections, SSL transfers, account type, and server location should all be weighed when choosing a Usenet provider. You can find a more detailed guide to all of our Usenet criteria here.
What you need to start using Usenet
To get started with Usenet, you’ll require three things: a subscription to a provider, a newsreader client, and an indexer/search engine. Some providers wrap all three of these things into one tidy package. If that’s not the case, free newsreaders and indexers are available online.
We’ve got a guide to walk you through the full Usenet setup process here.
The history of Usenet
Today’s Usenet is almost unrecognizable compared to when it first established in 1980. Today, it’s one of the most popular technologies for downloading videos and other digital media from other users, second only to BitTorrent. But it started out as a means for universities and experts in the field of technology to read and post messages in their community.
Usenet was a community for the brightest minds in tech. Prior to the proliferation of commercial internet to homes and businesses, many of the biggest advances in internet technology were first posted on Usenet. Tim Berners-Lee announced the launch of the World Wide Web in 1991. Linus Torvalds introduced the Linux Project in the same year. And in 1993, Marc Andreesen publicized the Mosaic browser, which popularized the World Wide Web and made it possible to add images to web pages.
Usenet was also the birthplace of many of today’s online ailments, including spam and trolls. Later on, it became a popular venue for the distribution of pirated and illicit content, ranging from popular movies and TV shows to child pornography.
This article will provide a layman-accessible chronology of the major developments in Usenet and its role alongside contemporaries, including ARPANET and the World Wide Web.
Related: A brief history of the internet
This article assumes you have at least some surface-level knowledge of Usenet and how it works. If this is your first time ever hearing of Usenet, we recommend you read our overview.
UUCP and A News
Jim Ellis and Tom Truscott first conceived of Usenet in 1979 while attending graduate school at Duke University. It was created to replace a BBS announcement program and create a connection with the University of North Carolina nearby. Its first wave of about 50 members came from a USENIX conference held at the University of Delaware in June 1980, where Duke’s Steve Daniel presented Usenet and invited attendees to join.
The first messages were sent via the A News, the original program for reading and serving Usenet newsgroups. A News, then just known as “news”, expanded on Unix’s “message of the day” feature that allowed system operators to display messages to users when they logged in. A News took this and combined it with UUCP (Unix-to-Unix Copy), a then-new protocol that allowed for remote transfer of files and messages between computers.
A News got its name because every message posted begins with the letter ‘A’ as a marker. Users read the messages right from the command line or when they logged in. They could add a message to be posted either on the local machine for other users to see when they logged in, or queue it to be sent over the entire network to anyone subscribed to a particular newsgroup. At first, Usenet was primarily used for making announcements, and the messages looked something like this:
Due to the bandwidth limitations of 1980s dial-up modems, A News was designed to be concise rather than robust. A News readers couldn’t respond over Usenet. Email responses were used instead. Users couldn’t skip messages, and posts weren’t threaded. The “path” noted in the image above shows the route that the message went through to arrive at its destination. “Bang paths” were very slow, sometimes taking up to a week to send news, and it was not uncommon for messages to be lost. These early iterations of email addresses were later superseded by the ‘@’ symbol we know today.
Bang paths are still used in Usenet messages today, but not for routing purposes.
By the end of its first year in operation, Usenet brought on 50 member sites from Bell Labs and universities like Reed College and University of Oklahoma.
NNTP and B News
Around the same time that Usenet first surfaced, ARPANET started expanding access to other protocols including TCP/IP, forming the foundation for today’s internet and the World Wide Web. But it also granted access to other protocols as well, including the UUCP protocol used by Usenet. By 1983, thousands of people joined Usenet, and over 500 UUCP hosts disseminated content through the network.
A News was soon made obsolete in 1983 by its successor, B News. B News was faster and more reliable, allowing up to 50 articles to be posted per day instead of just one or two. It was invented by Berkeley student Mark Horton and high school student Matt Glickman.
By 1984, only one year later, the number of hosts nearly doubled to 940, and more than 100 different newsgroups were up and running.
In 1986, two University of California engineers created the NNTP protocol. NNTP is used specifically for transporting Usenet articles between news servers so end users can read and post news. This allowed newsreader applications to run on personal computers connected to local networks instead of the server itself. Today, NNTP is still used by the majority of Usenet servers. NNTP allows Usenet articles to be distributed using TCP/IP, which ARPANET also adopted.
The Great Renaming
As Usenet grew in popularity, the current structure by which newsgroups were organized didn’t scale so well. All newsgroups were categorized into one of three categories: *net. **for unmoderated discussions, *mod. **for moderated discussions, or *fa. **for groups gatewayed from ARPANET. NNTP and B News lifted some of the software restraints, making it practical to reorganize newsgroups.
The reformation was largely led by an informal group of large site administrators known as the the backbone cabal. Started in 1983, this group made its news servers available 24 hours per day instead of the nighttime-only operations of their smaller counterparts. Usenet as a whole was decentralized and thus had no official leaders, but the backbone cabal acted as a respected authority in an otherwise chaotic community.
At the time, many doubted the existence of the cabal, but its influence allowed it to put forth the Great Renaming in 1987. This would fundamentally change the categories to comp.* (computers), misc.* (miscellaneous), news.* (Usenet and newsgroups), rec.* (recreation and entertainment), sci.* (science), soc.* (social), and talk.* (religion, politics, and other controversial subjects).
The new categories became known as the “Big Seven”, and they were open to everyone except for some of the moderated newsgroups within them. In 1995, after ISPs started offering Usenet access to the general public and Usenet became more mainstream among academics, .humanities* was added to the list, turning the Big Seven into the Big Eight. Under the new system, before any newsgroup could be created, the creator was required to submit a request for discussion (RFD) in the news.announce.newsgroups newsgroup. It was then considered in the news.groups.proposals newsgroup. Finally, the Big 8 Management Board (B8M8) had approve the proposal.
Not all of the old hierarchies were abolished; many such as the education-focused .k12* hierarchy remained in place.
Some speculated that the renaming took place to appease European networks that refused to pay for hosting high-volume but low-content newsgroups discussing controversial topics like religion and racism. These newsgroups were bundled together under .talk*. This made it easier for networks to exclude unwanted newsgroups from their Usenet servers. This did not sit well with many outspoken Usenet users who believed the Great Renaming would restrict their freedom of expression. Many topics were still off limits, including newsgroups about recreational drug use and sex.
Shortly after the Big Seven became the new status quo, another popular hierarchy was created: .alt* (alternative). Though it was never part of the Big Eight, .alt* became the go-to place for less academic and more extreme discussion that ISPs might not have tolerated in .talk*. It was free from the constraints and centralized control of the formal hierarchies in the Big Eight. Anyone could create an alt newsgroup. One popular FAQ on the subject joked that “alt” stood for Anarchists, Lunatics, and Terrorists. Today it is probably the most recognized hierarchy for the majority of Usenet users, largely thanks to the alt.binaries and alt.sex sections. More on those later.
The term “alt right” used to describe fringe groups with conservative ideologies did not originate on Usenet. The phrase’s mainstream adoption was likely accelerated, though, thanks to its familiarity among Usenet veterans who view it as an allusion to the conspiracy theorists and hate groups that congregated on Usenet long before 4Chan was ever conceived.
C News and InterNetNews
In 1987, University of Toronto staff members created the C News server package. They completely rewrote B News to make it faster and less buggy. C News continued to be used until the mid 1990s.
In 1991, C News was superseded by InterNetNews, another news server package that fully integrated NNTP functionality. INN is now the most commonly used server package and is still actively maintained.
ISPs join in
Until the 1990s, Usenet was the domain of colleges, universities, and tech research labs. Every year in September, freshman undergraduates would get their first taste, resulting in an annual wave of new Usenet users. They would either learn to love Usenet or hate it, and the numbers would eventually taper off for the remaining 11 months of the year.
Then, in September 1993, America Online (AOL) became the first major internet service provider to offer Usenet access to its users. Usenet became part of AOL’s larger marketing campaign to recruit new subscribers. This time, the numbers did not taper off, and Usenet became vastly more popular. The influx became known in the Usenet community as the “Eternal September.” Newsgroups sprung up in a similar fashion to today’s subreddits for every likeminded group of people tech savvy and patient enough to learn Usenet.
By the mid-90s, hosting Usenet servers became the norm for American ISPs, or they would at least offer subscribers an account with a third-party provider. Each host synchronized its content with everyone else, creating a uniform redundant network.
Usenet was still being used almost entirely for text-based interaction. Newsgroups were the precursors to today’s BBS forums (read: reddit), blogs, RSS feeds, and newsletters.
Sex and binaries
The .alt* hierarchy spawned a wide variety of newsgroups including those dedicated to specific celebrities and other aspects of pop culture. But the growth of two sections in particular around the mid-1990s signaled a paradigm shift that fundamentally changed Usenet forever, for better or worse.
A “binary” file refers to any non-text file. In the context of Usenet, binary typically refers to audiovisual media, including movies, TV shows, video games, software, and music. With internet service providers like AOL now hosting Usenet files, the ability for the network to transmit large amounts of data became feasible. This, in combination with early proliferation of home broadband networks and internet-connected home computers, kicked off Usenet’s transformation from a news and discussion aggregator to a file sharing network.
Alt.sex and alt.binaries later became the two most popular sections of Usenet. By October 1993, 8 percent of all Usenet users–3.3 million people–read news from alt.sex. Newsgroups posted pictures ranging from Playboy-style nudes to more niche fetishes and less socially acceptable topics, including bestiality and rape play.
New programs allowed binary files to be encoded in a Usenet-compatible way, making it more practical to distribute binary files. Files on Usenet often limit the amount of characters allowed in a post. To get around this limitation, new programs split up binary files into multiple smaller files that are later reassembled by the newsreader. In 2001, the yEnc file encoding was introduced, which cut data transfers down by 30 percent. Today’s Usenet users will no doubt recognize the encoding that’s now commonplace on binary files.
Alt.binaries distributed a wide range of other content but was particularly known for pirated, copyrighted material like movies, TV shows, software, and music. Even though bandwidth and server space were growing rapidly, Usenet hosts struggled to keep up with the large volume of data being uploaded and downloaded from their servers. Thus, binary retention limits were introduced, which would remove files after they reach a certain age to make room for new files.
Years before BitTorrent was invented, Usenet digressed from a platform for discussion to a decentralized hotbed of piracy and porn.
By the mid- to late-1990s, Usenet reached a turning point that many of its most fervent users agreed marked the beginning of a long decline. Those who remember it cite a few different reasons, but it was probably a combination of factors that led a once-burgeoning technology into relative obscurity.
The growth in active users peaked and started to drop off, but the costs of operating the network continued to increase due to the influx large binary files, many of which were either pirated media or pornography. When internet service providers realized that their subscribers could do without Usenet, they began cutting binary retention times and eventually Usenet access altogether. Abandoning Usenet was a win-win for ISPs that didn’t want to be held liable for the illegal content on their servers but did want to save money.
Another popular theory is that Usenet simply lost out to the World Wide Web. For a while, they were competing standards. Mosaic and Netscape browsers offered laymen a far easier and graphically pleasing medium to access content on the internet. Usenet stubbornly stood by its text-based interface, with no such graphical user interface ever taking hold among the larger community. As it turns out, command lines aren’t for everyone.
Usenet and NNTP have no identity authentication mechanism. This made it extremely difficult to root out spam, phishing scams, malware, forged headers, and general abuse.
In the early 2000s, Usenet’s unregulated network started to attract distributors of child pornography. Providers worked with law enforcement and allowed users to flag such content, but it was too little, too late for Usenet’s already shaky public image. AOL, the first ISP to bring Usenet into the mainstream fold, stopped offering access in 2005.
In 2008, New York State opened an investigation into child pornographers on Usenet. Then-Attorney General of New York Andrew Cuomo teamed up with Verizon Communications, Time Warner Cable, and Sprint to launch a public campaign to shut down sources of child pornography. Their solution was to cut off access to the entire .alt* hierarchy, which conveniently included the resource-hogging alt.binaries section. Tens of millions of internet users lost access to Usenet in one fell swoop, and many more ISPs followed. Once Usenet became associated with kiddie porn, no one would touch it with a 10-foot pole.
In 2010, Duke University–Usenet’s birthplace–decommissioned its Usenet server, citing high costs and low usage.
Independent news services
As alt.binaries grew in popularity, independent news services began cropping up to meet demand from users. These offered and continue to offer high quality service, fast downloads, and much longer retention times than those offered by ISPs. These include premium Usenet providers that still operate today, including EasyNews and Giganews among others.
When ISPs and universities started to jump ship, these services grew to fill in the gap. Unlike traditional hosts that offered Usenet access as part of a larger package, independent Usenet providers focused solely on Usenet. While they give users access to the entire network, they are typically used to download files from alt.binaries and other file sharing sections of the alt hierarchy.
By the mid-2000s, independent Usenet providers were the most popular means of accessing Usenet. But it was a win by default; ISPs and universities wanted nothing to do with it anymore.
Premium Usenet providers continued to up their game with bespoke newsreader clients, private encrypted access, and ever-growing retention times.
Usenet is still alive and active today, but it’s a mere shadow of its former glory. A handful of active discussion groups are still going strong, but the vast majority of Usenet subscribers use it to download binary files. The majority of Usenet servers are located in the US.
Usenet servers are also fairly common in countries that don’t comply with the Digital Millenium Copyright Act. This means they don’t have to remove files upon request from the copyright owner. While countries like the Netherlands and Germany have their own versions of these laws, Usenet providers there don’t have automated DMCA takedown mechanisms in place, and the requests are less frequent.
The average daily volume of data transferred on Usenet is actually higher than ever, according to premium provider Altopia. Usenet doesn’t have more users or active newsgroups, however. The increase consists of more and larger binary files being uploaded and downloaded, plus a huge amount of automated spam.
Pirated content from alt.binaries and pornography from alt.sex remain the most popular hierarchies. Child pornography is still a problem that Usenet struggles with.
An archive of non-binary Usenet posts back to 1981 is available on Google Groups. Deja News began archiving Usenet posts in 1995, creating a huge, searchable database. Google acquired the database in 2001, then added pre-1995 posts that were donated from a handful of universities and companies. Google continues to archive Usenet posts and provide a shared gateway to them, but some have criticized Google Groups, somewhat ironically, for its poor search functionality.
Altopia reports as of January 2017 Usenet users post 64.55 million times per day, for a total of 27.8 TB of daily volume, but it’s unclear how much of that is spam. The total number of newsgroups now stands at about 120,000, but only an estimated 20,000 are considered active.
Despite a long downward spiral, Usenet lives on. And as the World Wide Web becomes more fragmented and centralized by major corporations, some still see in Usenet the potential to reincarnate as a decentralized, free, and open internet made by users, for users.