Monday, August 11, 2008

The end of Usenet?

In this post I put up some plots with the posting and linking rates in our political Usenet data set. Would the political Usenet continue to decline at the current linear rate (according to the past 4 years, and using a rough linear regression in R), the post rate would approach 0 around July 2014. Here is the (unsmoothed) data for posting rates, and the completely unoptimized linear fit in R.

Of course, it's safe to say that regressing on 4 years of data and projecting it 6 years into the future will leave a huge margin of error-- even the fit "looks" like it should be a little steeper. (Smoothed data gives the endtimes to occur a year earlier, and I imagine more sophisticated time series analysis would project something completely different.) Errors involved in fitting aside, there's no telling what other sorts of things could happen between now and then to bring it back or (perhaps more likely) speed its decline.

5 comments:

Anonymous said...

Very nice article!

One question: how did you get the raw data ? The usenet articles from 2004 to 2008 ?

Regards,

Rick

Mary McGlohon said...

Rick,

Thanks for your comment.

Yes. The dataset we used was taking posts from around 200 political newsgroups (found by looking for substring 'polit'). From these groups we collected articles from 2004 to 2008. The graph reflects the number of posts per day.

Anonymous said...

Hi again (thanks for he quick reply)

My question was more about the datasource you used to get the usenet articles so far in the past ... do you use giganews or something ?


-rick

Anonymous said...

Hi again,

You might have missed my reply ... no hint of the data source you used ?

Cheers,

Rick

Mary McGlohon said...

Rick,

I apologize for not getting back to you. I am not sure whether I can disclose the data source at this point (I tend to play it safe on these things). I will let you know when I find out for sure that it's ok.

Thanks.