tag:blogger.com,1999:blog-36238998546680904742023-11-16T07:46:13.405-08:00The DataphilesA blog for data mining, machine learning, and other bits from stats, math, CS, and social sciences. And other stuff.Unknownnoreply@blogger.comBlogger46125tag:blogger.com,1999:blog-3623899854668090474.post-14684904855866886542010-08-01T18:43:00.000-07:002010-08-02T06:27:17.524-07:00Why can't we be journal-driven like real scientists?<span class="Apple-style-span" style="font-size:medium;"><span class="Apple-style-span" style="font-family:georgia;"><div><span class="Apple-style-span" style="font-size:small;">Disclaimer: </span><span class="Apple-style-span" style="line-height: 16px; "><i><span class="Apple-style-span" style="font-size:small;">I have been in the depths of thesis-draft writing, so I believe that around 60% of my angst is an extremely acute case of senioritis.</span></i></span><i><span class="Apple-style-span" style="font-size:small;"> </span></i><span class="Apple-style-span" style="font-family:Georgia, serif;"><span class="Apple-style-span" style=" line-height: 16px; font-family:georgia;"><i><span class="Apple-style-span" style="font-size:small;">However, the other 40% is founded on actual frustrations with academia, which I am working on articulating bit by bit.</span></i></span></span></div><div><span class="Apple-style-span" style="font-family:Georgia, serif;"><div><span class="Apple-style-span" style=" line-height: 16px; font-family:georgia;"><span class="Apple-style-span" style="font-size:small;"><br /></span></span></div><div><span class="Apple-style-span" style=" line-height: 16px; font-family:georgia;"><span class="Apple-style-span" style="font-size:small;">-------</span></span></div><div><span class="Apple-style-span" style=" line-height: 16px; font-family:georgia;"><span class="Apple-style-span" style="font-size:small;"><br /></span></span></div></span></div><div><span class="Apple-style-span" style="font-size:small;">I just got back from my last conference as a grad student. </span><span class="Apple-style-span" style="font-family:Georgia, serif;"><span class="Apple-style-span" style="font-family:georgia;"><span class="Apple-style-span" style="line-height: 16px; "><span class="Apple-style-span" style="font-size:small;">There were several papers I thought were really neat, some great talks, and lots of fun people to hang out with, which definitely made it worthwhile</span></span></span><span class="Apple-style-span" style=" line-height: 16px; font-family:georgia;"><span class="Apple-style-span" style="font-size:small;">. But conferences have their share of frustrations (which led to me tweeting, at one point, "</span><i><span class="Apple-style-span" style="font-size:small;">Academic conferences should provide more of a venue for punching people in the face</span></i><span class="Apple-style-span" style="font-size:small;">".) I think some of these frustrations would be less significant if CS were journal-driven like almost any other field.</span></span></span></div><div><span class="Apple-style-span" style="font-family:Georgia, serif;"><span class="Apple-style-span" style=" line-height: 16px; font-family:georgia;"><span class="Apple-style-span" style="font-size:small;"><br /></span></span></span></div><div><span class="Apple-style-span" style="font-family:Georgia, serif;"><span class="Apple-style-span" style=" line-height: 16px; font-family:georgia;"><span class="Apple-style-span" style="font-size:small;">This has been well articulated by Lance Fortnow in "Time for Computer Science to Grow Up" (</span><a href="http://www.cs.uchicago.edu/~fortnow/papers/growup.pdf"><span class="Apple-style-span" style="font-size:small;">pdf</span></a><span class="Apple-style-span" style="font-size:small;">). He reasons that the quality of publications would increase if we moved completed research to journals, and that using conferences as endpoints simply detracts from the main purpose of conferences: to bring people together. I'm not sure about the former, since others have pointed out to me that some flaws he mentions (biased reviewers, overspecialization, sloppy pre-deadline rushes, publish-or-perish, etc.) exist regardless of publication model. But, I think there are some things to be said for modifying the model just so conferences will be more fun.</span></span></span></div><div><span class="Apple-style-span" style="font-family:Georgia, serif;"><span class="Apple-style-span" style=" line-height: 16px; font-family:georgia;"><span class="Apple-style-span" style="font-size:small;"><br /></span></span></span></div><div><span class="Apple-style-span" style="font-family:Georgia, serif;"><span class="Apple-style-span" style=" line-height: 16px; font-family:georgia;"><span class="Apple-style-span" style="font-size:small;">And I'm way more concerned with how much fun our field is than its publication quality.</span></span></span></div><div><span class="Apple-style-span" style="font-family:Georgia, serif;"><span class="Apple-style-span" style=" line-height: 16px; font-family:georgia;"><span class="Apple-style-span" style="font-size:small;"><br /></span></span></span></div><div><span class="Apple-style-span" style="font-family:Georgia, serif;"><span class="Apple-style-span" style=" line-height: 16px; font-family:georgia;"><span class="Apple-style-span" style="font-size:small;">Most conference talks suck. It takes a huge amount of time to make a quality presentation, and there's little incentive to do so. Unless your talk is just plain offensive, the worst outcome for a thrown-together presentation is that people will forget it. For posterity, there's a good chance the line on your CV for the publication is all that matters. </span></span></span><span class="Apple-style-span" style="font-family:Georgia, serif;"><span class="Apple-style-span" style=" line-height: 16px; font-family:georgia;"><span class="Apple-style-span" style="font-size:small;">Yet, if you get a paper in a conference you're typically required to give a talk, so we end up with a lot of mediocrity. </span></span></span><span class="Apple-style-span" style="line-height: 16px; "><span class="Apple-style-span" style="font-size:small;">(</span><a href="http://behind-the-enemy-lines.blogspot.com/2010/07/peer-reviewing-for-oral-presentations.html"><span class="Apple-style-span" style="font-size:small;">Panos Iperiotis</span></a><span class="Apple-style-span" style="font-size:small;"> proposed having peer-review for talks, which sounds great in theory, but good luck finding enough people for </span><i><span class="Apple-style-span" style="font-size:small;">that</span></i><span class="Apple-style-span" style="font-size:small;"> review committee.)</span></span></div><div><span class="Apple-style-span" style="line-height: 16px; "><span class="Apple-style-span" style="font-size:small;"><br /></span></span></div><div><span class="Apple-style-span" style="line-height: 16px; "><span class="Apple-style-span" style="font-size:small;">Poster conferences like NIPS avoid a lot of that, but conference attendance is expensive. And it's a shame that people should have to pick what publication to submit to based on where they can get travel visas.</span></span></div><div><span class="Apple-style-span" style="line-height: 16px; "><span class="Apple-style-span" style="font-size:small;"><br /></span></span></div><div><span class="Apple-style-span" style="line-height: 16px; "><span class="Apple-style-span" style="font-size:small;">Journal-driven fields treat conferences like our (less-well-attended) workshops, which seems more appropriate. CS conference talks leave little room for discussion. Difficult questions are typically perceived as an attack, tabled with "That's a good question! Let's discuss this offline". After all, the paper has already been published in essentially final form-- what's the point in arguing with the authors except to make yourself look smart? Furthermore, if one </span><i><span class="Apple-style-span" style="font-size:small;">does</span></i><span class="Apple-style-span" style="font-size:small;"> have a significant issue with a paper that they'd like to address in the public forum, there's no "letters to the editor" section as in several journals. There's only Open Mic Night, and most of the audience is either checking email or leaving to go see a talk in a different track.</span></span></div><div><span class="Apple-style-span" style="font-family:Georgia, serif;"><span class="Apple-style-span" style=" line-height: 16px; font-family:georgia;"><span class="Apple-style-span" style="font-size:small;"><br /></span></span></span></div><div><span class="Apple-style-span" style="font-family:Georgia, serif;"><span class="Apple-style-span" style=" line-height: 16px; font-family:georgia;"><span class="Apple-style-span" style="font-size:small;">I'm sure there's a joke about how the super-introverted CS crowd wouldn't go to conferences unless they had to. But o</span></span></span><span class="Apple-style-span" style="line-height: 16px; "><span class="Apple-style-span" style="font-size:small;">verall, I think the conference as the end point of publication creates a high-pressure situation that detracts from the open forum it should be, and makes attendees less sociable.</span></span></div><div><span class="Apple-style-span" style="font-family:Georgia, serif;"><span class="Apple-style-span" style=" line-height: 16px; font-family:georgia;"><span class="Apple-style-span" style="font-size:small;"><br /></span></span></span></div><div><span class="Apple-style-span" style="line-height: 16px; "><span class="Apple-style-span" style="font-size:small;">In lieu of changing the model, I would advocate each conference having a Punch-People-In-The-Face Plenary Melee.</span></span></div></span></span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-3623899854668090474.post-37783086744886683942009-09-26T13:51:00.000-07:002009-09-26T14:43:20.422-07:00Machine Learning Protest at G20<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUyKPfp16X-KDjaq1KSGlXBaxeDlIiF1iGsEu3pMSJcMMb4jZnQlsRXfoGY2ab5c0fzcDs9fZir2w6Zbdn_Oqcp-VfvH_MyS5-ifyn4RK-mDYS8r_eDgwcx-iz-MhBHhkd8K4vkRElZ2P-/s1600-h/protest2.jpg"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 176px; height: 189px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUyKPfp16X-KDjaq1KSGlXBaxeDlIiF1iGsEu3pMSJcMMb4jZnQlsRXfoGY2ab5c0fzcDs9fZir2w6Zbdn_Oqcp-VfvH_MyS5-ifyn4RK-mDYS8r_eDgwcx-iz-MhBHhkd8K4vkRElZ2P-/s320/protest2.jpg" alt="" id="BLOGGER_PHOTO_ID_5385882391277206450" border="0" /></a>I have returned to the blogosphere to report on our successful voicing of machine learning concerns at the G20 People's March yesterday.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://farm4.static.flickr.com/3422/3953914015_fb11e85da2.jpg"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer; width: 441px; height: 294px;" src="http://farm4.static.flickr.com/3422/3953914015_fb11e85da2.jpg" alt="" border="0" /></a><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br />(Top: "Support Vector Machines," "Repeal Power Laws," "End Duality Gap," "MapReduce, MapReuse, MapRecycle: Green Data Processing." Bottom: "Bayesians Against Discrimination"," "Free Variables," "Ban Genetic Algorithms")<br /><br /><br />Several CMU SCS grad students and postdocs gathered at CMU and walked to Oakland where the march was to begin. As we carried our signs the LOL:Puzzlement ratio decreased (but by no means disappeared) as the distance from CMU increased. Then we marched with the crowd towards dahntahn.<br /><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://farm4.static.flickr.com/3489/3953914237_a74cc0ba3e.jpg"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer; width: 356px; height: 237px;" src="http://farm4.static.flickr.com/3489/3953914237_a74cc0ba3e.jpg" alt="" border="0" /></a><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br />John Oliver of the Daily Show showed up.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://farm3.static.flickr.com/2602/3953574449_70f1441b94.jpg"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer; width: 180px; height: 233px;" src="http://farm3.static.flickr.com/2602/3953574449_70f1441b94.jpg" alt="" border="0" /></a><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br />At some point we realized we were walking in front of the United Steelworkers team, who were all wearing the same hard hats as me (my friend having snagged mine from the USW booth at Netroots Nation last month). Here I am with a USW comparing causes.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://farm3.static.flickr.com/2509/3954694814_ba77e80705.jpg"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer; width: 284px; height: 189px;" src="http://farm3.static.flickr.com/2509/3954694814_ba77e80705.jpg" alt="" border="0" /></a><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br />All in all it was a success. For full set of pictures, go <a href="http://www.flickr.com/photos/30686429@N07/sets/72157622330082619/">HERE</a>.<br /><br />For publications related to machine learning activism, see "Data Mining Disasters: A Report" <a href="http://www.cs.cmu.edu/%7Emmcgloho/pubs/accidents-sigbovik08.pdf">[pdf]</a>, from <a href="http://sigbovik.org/2008/">SIGBOVIK 2008</a>; and "MapReuse and MapRecycle: Two More Frameworks for Eco-Friendly Data Processing" <a href="http://www.cs.cmu.edu/%7Emmcgloho/pubs/mapreduce.pdf">[pdf]</a>, of <a href="http://sigbovik.org/2009/">SIGBOVIK 2009</a>.Unknownnoreply@blogger.com25tag:blogger.com,1999:blog-3623899854668090474.post-79429653090861614332008-09-07T20:29:00.000-07:002008-09-07T22:01:59.273-07:00The popular get richer<span style="font-style: italic;">Pardon the hiatus; I have been busy at KDD and moving back to Pittsburgh (which included a 4-day scenic drive across the country).</span><br /><br />In the spirit of the season I've been looking at a large dataset of campaign donations, from 1980 to 2006. This data is free to the public from the FEC; I've parsed and made it available <a href="http://www.cs.cmu.edu/%7Emmcgloho/fec/data/fec_data.html">on my website</a>.<br /><br />One can form a bipartite graph of committees (such as the Underwater Basketweavers' Political Action Committee) and candidates (such as Abraham Lincoln). Individual donations are all filtered through committees (usually a candidate has one or several designated committees), so the organization-candidate graph is the best way to measure donations to specific candidates.<br /><br />A surprising observation in our <a href="http://www.cs.cmu.edu/%7Emmcgloho/pubs/butterfly-kdd08.pdf">KDD paper</a> was the "fortification effect". First, if one takes the number of unique edges added to the graph (that is, the number of interactions between orgs and candidates) and compares with the total weight of the graph (that is, the total $ donated), one finds super-linear behavior. That is, the more unique donor-candidate relationships, the higher the average check becomes. The power law exponent in the org-cand graph was 1.5. (This also holds for $ vs nonunique edges, or number of checks, with exponent 1.15).<br /><br />Even more interestingly, if one looks closer into the individual candidates, similar behavior emerges. <span style="font-weight: bold;">The more donors a candidate has, the higher the average amount received from a donor becomes.</span> The plot below shows money received from candidates vs. number of donor organizations.<br /><br /><img src="http://www.cs.cmu.edu/%7Emmcgloho/images/obs4.png" width="600" /><br /><br />Each green point represents one candidate, with the y-axis being the total money that candidate has received, and the x-coordinate being the number of donating organizations. The lower points represent the median for edge-intervals, with upper quartile error bars drawn. The red line is the power law fit-- here we have super-linear behavior between number of donors and the amount donated (with exponent 1.17). And again, the same is true for non-unique donations-- <span style="font-weight: bold;">the more checks, the higher the average check. </span><br /><br />Again, this does not include the 2008 data. I hear that Obama's donation patterns are different (lots of little checks, they tell me), but haven't confirmed this yet.Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-3623899854668090474.post-30141861021981455102008-08-27T18:00:00.000-07:002008-08-28T12:10:36.435-07:00KDD 2008 <3's social networksKDD this year put a lot of focus on social networks research. In addition to the long-running <a href="http://webmining.spd.louisville.edu/webkdd08/">WebKDD,</a> there was <a href="http://workshops.socialnetworkanalysis.info/SNAKDD2008/">SNA-KDD</a> (produced "Snack DD". Or at least it should be.) The <a href="http://www.cs.fiu.edu/%7Etaoli/kdd08-workshop/workshop.htm">Matrix and Tensor Methods</a> workshop also included a paper featuring graph mining, and of course matrix methods in general are important for SNA. There were also two relevant <a href="http://www.sigkdd.org/kdd2008/tutorials.html">tutorials</a> I plan look over, now that I have the proceedings: Neville and Provost's Predictive Modeling with Social Networks, and Liu and Agrawal's "Blogosphere: Research Issues, Applications, and Tools."<br /><br />And so far I've only listed SNA-related things going on first day. The Social Networks research track session was <span style="font-style: italic;">completely</span> full: all seats taken, people crammed into the aisles, and overflowing into the hall. It was unfortunate that there were several people who just couldn't get in, but it was a lot of fun for the speakers to have such a great audience. There was also a panel on social networks, featuring Backstrom, Faloutsos, Jensen, and Leskovec. The conference took a cue from the overflowing session and switched it to a bigger room. (Unfortunately, it was scheduled at the same time as some other regular sessions that didn't receive the audience they might have were they only paired with other sessions.) There was an industry track session also devoted to social networks, and other relevant papers were in the Text Mining and Graph Mining sessions.<br /><br />It's pretty exciting that social networks research is getting so much more attention, even in the last year. It will be interesting to see how long it lasts, and what sort of Big Questions get answered.<br /><br />Time to board. It looks like the plane back to Seattle is mostly full of a different kind of geek: <a href="http://www.pennyarcadeexpo.com/">PAX</a> attendees.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-3623899854668090474.post-82906495421550656912008-08-19T17:23:00.001-07:002008-08-19T18:03:50.825-07:00Butterflies in VegasI'd like to take this opportunity to self-promote our talk, "<a href="http://www.cs.cmu.edu/%7Emmcgloho/pubs/butterfly-kdd08.pdf">Weighted Graphs and Disconnected Components</a>", at <a href="http://www.sigkdd.org/kdd2008/">KDD 2008</a> next week in Las Vegas. This is work from CMU, with <a href="http://www.andrew.cmu.edu/%7Elakoglu">Leman Akoglu</a> and <a href="http://www.cs.cmu.edu/%7Echristos">Christos Faloutsos</a>. In it we look at some commonly overlooked features of social networks: "disconnected components", or the components that are not connected to the largest connected component; and weighted edges, edges formed through repeated or valued interactions. We also propose the "Butterfly model" to generate graphs that match some of these new properties, along with previously-established graph properties.<br /><br />For those planning to attend KDD, it will be the third in the <a href="http://www.sigkdd.org/kdd2008/sessions.html#r2">Social Networks Research</a> session, on Monday morning (with a poster Monday evening). Even if I haven't convinced you to attend our talk, you will want to see the other talks in the session (one of which is reviewed by Akshay Java <a href="http://socialmedia.typepad.com/blog/2008/08/structure-of-information-pathways-in-social-networks.html">here</a>). They'll include the usual suspects from Yahoo! Research and Cornell, plus a paper analyzing of Sprint's call network. It should be a fascinating couple of hours!Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-3623899854668090474.post-27325322880724879122008-08-18T11:28:00.000-07:002008-08-18T13:26:24.558-07:00Usenet vs blog linking in the 2004 electionAs a historical study, I've taken a subset of the political Usenet links and compared to blog links in the same time period. I took links from October-November 2004 and made "A lists" for both political and conservative blogs, and compared with the A-lists found by <a href="http://www.blogpulse.com/papers/2005/AdamicGlanceBlogWWW.pdf">Adamic and Glance in 2004</a>.<br /><br /><div style="text-align: center;"><span style="font-weight: bold;">Conservative A-List</span></div><br /><img src="http://www.cs.cmu.edu/%7Emmcgloho/images/con-blogs.png" /><br /><br /><div style="text-align: center;"><span style="font-weight: bold;"><br />Liberal A-List</span><br /></div><br /><img src="http://www.cs.cmu.edu/%7Emmcgloho/images/lib-blogs.png" /><br /><br />I did little in the way of pruning the list, while Adamic and Glance were careful to only include traditionally-formatted blogs, but I have removed ones they explicitly mentioned omitting. As in their study, I've left off <span style="font-style: italic;">drudgereport </span>(originally #2 in conservative) and <span style="font-style: italic;">democraticunderground </span>(originally #1 in liberal), as they were not "weblogs" in the traditional sense-- <span style="font-style: italic;">drudgereport </span>is an aggregator and <span style="font-style: italic;">democraticunderground </span>is a message board. <span style="font-style: italic;">freerepublic.com </span>is also message board-like, so that might have warranted removal from the list as well, but it was not mentioned in the paper. <span style="font-style: italic;">Bluelemur </span>also claims to be the liberal version of <span style="font-style: italic;">drudgereport</span>, (<span style="font-weight: bold;">ETA</span>: and realclearpolitics is another news aggregator) so those might also have been eliminated from the original study. <span style="font-style: italic;">Gadflyer.com</span>, #2 in Liberals, is no longer in existence, which is interesting in itself, considering its popularity just a few years ago. But it is also notably missing from the Blog A-list, either by the numbers or by classification.<br /><br />It is perhaps curious that blog-message boards <span style="font-style: italic;">democraticunderground </span>and <span style="font-style: italic;">freerepublic </span>have topped both lists in Usenet, which is closer to message board format. I wonder where they actually ranked on the pre-pruned blog list.<br /><br />My overall impression from looking at this data is that Usenet is "edgier"; we more commonly see conspiracy theories and wingers getting a lot of attention. Given that, and the apparent popularity of message-board-like blogs, I wonder if we could consider the Usenet to be even more "democratic" (or "ruled by the mob", to be more cynical) than blogs.Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-3623899854668090474.post-33696472637608908622008-08-14T09:37:00.000-07:002008-08-14T10:36:53.541-07:00Preferential Installation of Facebook Apps [SIGCOMM WOSN]I'm reading over the proceedings of SIGCOMM's <a href="http://conferences.sigcomm.org/sigcomm/2008/workshops/wosn/">Workshop On Social Networks</a>, which is in Seattle next Monday.<br /><br />Minas Gjoka, Michael Sirivianos, Athina Markopoulou, and Xiaowei Yang, a team of authors at UCI, wrote a paper, <a href="http://conferences.sigcomm.org/sigcomm/2008/workshops/wosn/papers/p31.pdf">Poking Facebook: Characterization of OSN Applications</a>, which looks at data from Facebook application installation and use.<br /><br />First, they seem to have gotten a pretty successful crawl, which is saying something since Facebook is pretty selfish with data. Here is a PDF of application installation, both according to facebook stats and their crawled dataset, which match up pretty well:<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiscniPa-l5-bI2duiqHsyzRl7u1JK5DXoPO8buUOYBCUqN6H19WOnHjjUIjPBmHDnCT_YbSRJHIVhAoagzl-_uiFCHvU6Zt4Moz7WILRpqTT0gBeCk0fdj-mJM2NqMpCznO6G78CZJxQaV/s1600-h/sigcomm-facebook-1.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiscniPa-l5-bI2duiqHsyzRl7u1JK5DXoPO8buUOYBCUqN6H19WOnHjjUIjPBmHDnCT_YbSRJHIVhAoagzl-_uiFCHvU6Zt4Moz7WILRpqTT0gBeCk0fdj-mJM2NqMpCznO6G78CZJxQaV/s400/sigcomm-facebook-1.png" alt="" id="BLOGGER_PHOTO_ID_5234416691497286642" border="0" /></a><br />They also modeled the histogram of installed-apps-per-user as preferential, running a simulation with "users as bins" and different apps as "different colored balls", iteratively assigning balls to bins. For instance, saying that 100 users have installed the Zombies application, would translate to "gray balls appear in 100 bins".<br /><br />For each iteration, one goes through each "ball" (application installation), starting with the "most popular color" (application with the most installations). For each ball one then assigns an additional "bin that doesn't already contain that color" (picks a new user that hasn't already installed the app) according to a probability:<br /><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhg_jcn8BZLF_iLn1AOyjKvuF6YnrI-0g64_2f3hmH9CnYqyfHSCFHBiNv8qXlNUctaTGH8f9x1X16w1nZE4FoauRYXPArP9rgdbFjHDDqYy6oynm_S0xvSn4ixsepzNBvd3lduWBI-hfjk/s1600-h/sigcomm-facebook-3.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhg_jcn8BZLF_iLn1AOyjKvuF6YnrI-0g64_2f3hmH9CnYqyfHSCFHBiNv8qXlNUctaTGH8f9x1X16w1nZE4FoauRYXPArP9rgdbFjHDDqYy6oynm_S0xvSn4ixsepzNBvd3lduWBI-hfjk/s400/sigcomm-facebook-3.png" alt="" id="BLOGGER_PHOTO_ID_5234418167369308242" border="0" /></a><br />Where <span style="font-style: italic;">balls(i) </span>is the number of applications a user <span style="font-style: italic;">i </span>has installed, and <span style="font-style: italic;">B</span> is the set of users that hasn't already installed the application. <span style="font-style: italic;">init</span> is a parameter to moderate the initial activity, and rho is the preferential exponent, chosen in simulations to be 1.6. In the end you get a sort of heavy-tailed behavior, with most users installing a couple apps and a few who go nuts with application installs. It fits pretty well:<br /><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlGTy01yTqVL6YvpcfDyiHxHmZBTUuAnNwRRkf_y4Iu6qdVKdWITE-gpHLUZ34hxd2WIb2lxK4ir-I1DwlGiwJWcYR2VACTQVBavkF5t0IZ8TiaX2mjqN9c4wGNN_6VwIwkV-1r41na7AN/s1600-h/sigcomm-facebook-2.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlGTy01yTqVL6YvpcfDyiHxHmZBTUuAnNwRRkf_y4Iu6qdVKdWITE-gpHLUZ34hxd2WIb2lxK4ir-I1DwlGiwJWcYR2VACTQVBavkF5t0IZ8TiaX2mjqN9c4wGNN_6VwIwkV-1r41na7AN/s400/sigcomm-facebook-2.png" alt="" id="BLOGGER_PHOTO_ID_5234417366608070898" border="0" /></a><img src="file:///C:/Users/t-marymc/AppData/Local/Temp/moz-screenshot-2.jpg" alt="" /><br />One of the fun parts of these sort of data is the outliers-- the users who go nuts on something. (Netflix users rating thousands of movies, etc.) It looks like in the crawled data there are a few users with 500 apps installed!<br /><br />In the paper there is also fit to the "coverage of applications"-- that is, how many of the ranked apps we need to go through before we have all the users with one of those apps installed, and it appears the simulation reaches coverage a little too quickly, so perhaps the most popular applications are taking too many users in the simulation.<br /><br />What's somewhat surprising to me is that this isn't at all based on the behavior of a user's friends, but of the entire Facebook network at large. I suspect that in reality that does govern user behavior, but for large-scale patterns one can overlook it. This might be different for actually modeling how an application catches on. (Using other features like network effects are listed as "future work" for the authors in refining the model.)<br /><img src="file:///C:/Users/t-marymc/AppData/Local/Temp/moz-screenshot.jpg" alt="" /><img src="file:///C:/Users/t-marymc/AppData/Local/Temp/moz-screenshot-1.jpg" alt="" />Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-3623899854668090474.post-57088833866417273012008-08-12T19:05:00.000-07:002008-08-13T11:21:44.822-07:00Shared authors in the political UsenetMy apologies if this image messes with your RSS feed readers. It doesn't show up well on the main page, so go <a href="http://www.cs.cmu.edu/%7Emmcgloho/images/shared-authors-mod.png">here</a> for full view.<br /><br />Using Marc Smith's .Netmap plug-in for excel, I visualized some data I had. These are shared authorships of political Usenet groups, based on <a href="http://en.wikipedia.org/wiki/Jaccard_index">Jaccard coefficient</a> (similar to cosine similarity). A thin edge indicates a coefficient > 0.3, a thick edge indicates >0.5.<br /><br />The alt.politics.* and talk.politics groups were a tangled mess that's nearly a clique, but there is some interesting behavior with local groups. In the top left are the Canadian local groups. Quebec's qc.politique doesn't appear at all in this graph (nodes are only visible if there is an edge associated), probably due to language barrier. Then, we have Saskatchewan and Manitoba connected with a thick edge, and British Columbia, Alberta, and Ontario connected with thick edges. Only the latter group is connected with can.politics, the general Canada group. Looking at the Canadian map it isn't regional since ONT is east of SK and MAN. However, there is something that does correlate the groups: <a href="http://en.wikipedia.org/wiki/List_of_Canadian_provinces_and_territories_by_population">population density</a>. The group of three has a higher population, and a significantly higher population density, than the group of two. What comes with that is a higher-traffic local group, and more authors with which to share with even larger groups-- giving a higher coefficient.<br /><img src="http://www.cs.cmu.edu/%7Emmcgloho/images/shared-authors-mod.png" /><br /><br />The same thing may be happening with the US local groups, too. I've circled the "connected" US groups-- that is, the ones that share lots of authors with the alt.politics.* group. What's interesting is that these high-traffic groups form a bridge between alt.politics.* and the other local US groups. Several of these statewide, lower-traffic groups share authors amongst themselves, but with only a couple exceptions, they don't venture outside the local politics sphere. And again, there are some that don't show up here (most notably Virginia and Maryland-- I would imagine their nearest neighbor would be dc.politics, but they didn't have enough volume to get a high share-rate).<br /><br />Just some cool-looking effects of the Jaccard index. I think another interesting way to visualize this might be to use a digraph, with an arrow from A to B if "p% of A's authors post in B". I bet that would get Virginia and Maryland to show up.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-3623899854668090474.post-56118298334148337972008-08-12T08:44:00.000-07:002008-08-12T09:18:52.540-07:00Disney and CMU collaborate on graphics, pixie dust analysisThe newest resident to the Collaborative Innovation Center (CIC) on the CMU campus is the <a href="http://www.cs.cmu.edu/news/releases/disney.html">Disney Research Pittsburgh Lab</a>, joining Google, Intel, and Apple Pittsburgh labs. This is, of course, pretty sweet.<br /><br />One thing CMU has to navigate is a relative lack of nearby tech industry compared to schools on the coasts. Industry collaborations have not been a huge problem, as they can be done over long distance and CMU is very good at encouraging them through internships/sabbaticals. (Of course, having folks next door at the CIC makes things easier.) However, when it comes to tech-industry couples, it's always good to have more potential workplaces. Sometimes CMU doesn't have two faculty positions, or two grad student positions, etc. For more admin-type analysis, see post from CSD head Peter Lee at <a href="http://www.csdhead.cs.cmu.edu/blog/2008/08/12/disney-to-open-research-labs-on-cmu-campus/">CSDiary</a>.<br /><br />CMU will collaborate with Disney Research Pittsburgh primarily on autonomous systems, graphics, and other entertainment technologies. Collaborations thus far have proved difficult, however, due to Donald Duck's frequent temper tantrums and insertion of nonsensical paragraphs into the text of papers. Also, Tigger keeps tearing up the lab equipment.Unknownnoreply@blogger.com6tag:blogger.com,1999:blog-3623899854668090474.post-45809839109487076792008-08-11T11:58:00.000-07:002008-08-13T11:22:02.191-07:00The end of Usenet?In <a href="http://dataphiles.blogspot.com/2008/07/what-has-been-happening-to-political.html">this post</a> I put up some plots with the posting and linking rates in our political Usenet data set. Would the political Usenet continue to decline at the current linear rate (according to the past 4 years, and using a rough linear regression in R), the post rate would approach 0 around July 2014. Here is the (unsmoothed) data for posting rates, and the completely unoptimized linear fit in R.<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjt37zlo5KT-O2fvRgPwlgD_8TNmJ9FSGL0C3KCLNQgG6ohGwuJojOw5YL-It0i1IEkzpCg8U5nTQErlnaZmFNxyTOv2iSDEIvL5wZ_1F_s0DacGc5jjGA1QKuM3i8v5NKwOm2gCG3AEEJF/s1600-h/posts-regression.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjt37zlo5KT-O2fvRgPwlgD_8TNmJ9FSGL0C3KCLNQgG6ohGwuJojOw5YL-It0i1IEkzpCg8U5nTQErlnaZmFNxyTOv2iSDEIvL5wZ_1F_s0DacGc5jjGA1QKuM3i8v5NKwOm2gCG3AEEJF/s400/posts-regression.png" alt="" id="BLOGGER_PHOTO_ID_5233339214633963986" border="0" /></a><br />Of course, it's safe to say that regressing on 4 years of data and projecting it 6 years into the future will leave a huge margin of error-- even the fit "looks" like it should be a little steeper. (Smoothed data gives the endtimes to occur a year earlier, and I imagine more sophisticated time series analysis would project something completely different.) Errors involved in fitting aside, there's no telling what other sorts of things could happen between now and then to bring it back or (perhaps more likely) speed its decline.Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-3623899854668090474.post-90084723922207371652008-08-09T00:25:00.000-07:002008-08-13T11:22:25.729-07:00What is Eliza crossed with a Magic 8 ball, times a billion?I just got my weekend entertainment from <a href="http://bossy.appspot.com/">http://bossy.appspot.com</a>. It's an "ask" app inspired by <a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.3497">Ask MSR</a>, a paper written 7 years ago for TREC. (I'm not sure the authors ever intended or expected such a thing, but there it is.)<br /><br />For <a href="http://zooie.wordpress.com/2008/08/04/yahoo-boss-google-app-engine-integrated/">50 lines of API code</a>, it's pretty impressive-- and when it's wrong it's usually entertaining. It does well on short and simple word-association queries, like "<a href="http://bossy.appspot.com/qa?query=Who+is+Batman">Who is Batman</a>?" or "<a href="http://bossy.appspot.com/qa?query=what+is+xkcd%3F">What is xkcd?</a>", and seems to do reasonably well on easily-searchable names such as my own.<br /><br />It even has some political opinions. Try "<a href="http://bossy.appspot.com/qa?query=who+is+the+worst+president+in+history%3F">Who is the worst president in history</a>?", "<a href="http://bossy.appspot.com/qa?query=who+lost+the+2004+election%3F">Who lost the 2004 election?</a>", or "<a href="http://bossy.appspot.com/qa?query=What+is+the+United+States">What is the United States</a>?" and you'll get some cynical answers. It's also <a href="http://bossy.appspot.com/qa?query=who+would+win%2C+iran+or+the+usa%3F">pacifist</a>, at least for certain queries.<br /><br />It tends to get snarky when asked other binary queries. I was chagrined when I asked it "<a href="http://bossy.appspot.com/qa?query=which+is+better%2C+cmu+or+mit%3F">Which is better, CMU or MIT?</a>" It also defects when asked to decide between <a href="http://bossy.appspot.com/qa?query=which+is+better%2C+microsoft+or+google%3F">Microsoft and Google</a>, or between <a href="http://bossy.appspot.com/qa?query=which+is+better%2C+OU+or+OSU%3F">OU and OSU</a>. However, does have a preference with respect to the <a href="http://bossy.appspot.com/qa?query=bayesians+or+frequentists%3F">statistics cults</a>.<br /><br />Alas, it does not seem to be immune to <a href="http://bossy.appspot.com/qa?query=what+is+better+than+google%3F">spam</a>.Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-3623899854668090474.post-44411320659212915012008-08-07T13:23:00.000-07:002008-08-13T11:22:41.898-07:00The male-female demographic in social media<a href="http://www.mikeonads.com/">Mike on Ads</a> has a cool <a href="http://www.mikeonads.com/2008/07/13/using-your-browser-url-history-estimate-gender/">script </a>to infer whether a user is male or female, based on browser history. If you have it analyze your history, it will give you a list of the sites you visited and the corresponding male:female ratio. He got the sites to poll from <a href="http://www.quantcast.com/">quantcast</a>, but I can't tell if the demographics came from there as well. The numbers seem to be different when I plug them in, so I'm guessing he either used more/older data than what's currently up, or got it elsewhere.<br /><br />Here are the ratios for various social media I plugged in, in order from "most male" to "most female":<br /><br /><span style="font-weight: bold;">Site M:F ratio</span><br />Digg 1.56<br />Flickr 1.15<br />Feedburner 1.11<br />Worldofwarcraft 1.08<br />Blogger 1.06<br />Youtube 1<br />Last.fm 0.96<br />Linkedin 0.94<br />Pandora 0.9<br />Facebook 0.83<br />Myspace 0.74<br />Livejournal 0.68<br /><br />Twitter and Wikipedia didn't seem to be in the feature set. However, straight form quantcast it seems Twitter's ratio is 0.97 and Wikipedia's is 1.07. Quantcast also lists a ton of other demographic info, which is interesting to look at.Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-3623899854668090474.post-81345363798350155402008-08-05T19:49:00.000-07:002008-08-05T21:30:09.898-07:00How our brains deal with large numbersVia <a href="http://www.stat.columbia.edu/%7Ecook/movabletype/archives/2008/08/a_natural_log_o.html">Andrew Gelman</a>, a recent <span style="font-style: italic;">Science</span> article claims that humans innately use a logarithmic scale.<br /><blockquote></blockquote><blockquote>When asked<sup> </sup>to point toward the correct location for a spoken number word<sup> </sup>onto a line segment labeled with 0 at left and 100 at right,<sup> </sup>even kindergarteners understand the task and behave nonrandomly,<sup> </sup>systematically placing smaller numbers at left and larger numbers<sup> </sup>at right. They do not distribute the numbers evenly, however,<sup> </sup>and instead devote more space to small numbers, imposing a compressed<sup> </sup>logarithmic mapping. For instance, they might place number 10<sup> </sup>near the middle of the 0-to-100 segment.</blockquote>(Full text <a href="http://www.sciencemag.org/cgi/content/full/320/5880/1217">here</a>, SciAm report <a href="http://www.sciam.com/article.cfm?id=a-natural-log&SID=mail&sc=emailfriend">here</a>)<br /><br />When I was a little kid my dad helped me "count to a million" using log scale (1,2,3...10,20,...100,200,...). Even then it seemed intuitive. I knew that there were increasingly more numbers in between counts as it got higher, and I felt I was "cheating" by skipping them, but I did not understand how long it truly would have taken if we'd counted all the numbers in between (I probably would have guessed it'd have taken hours, rather than days).<br /><br />It's not that people cannot grasp large numbers-- they just have trouble converting back to a linear scale. :-)Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-3623899854668090474.post-88063234613774188512008-08-04T18:24:00.001-07:002008-08-13T11:22:58.485-07:00Memeorandum : Scandal : : Techmeme : ?A few days ago, I <a href="http://dataphiles.blogspot.com/2008/07/scandal-sells.html">posted </a>that the top "most discussed" links on <a href="http://www.memeorandum.com/">Memeorandum</a> were related to scandal, violence, or both. <a href="http://socialmedia.typepad.com/blog">Akshay</a> suggested I do the same for <a href="http://www.techmeme.com/">Techmeme</a>. His bid on the #1 discussed story was the Microsoft-Yahoo merger. For updates since September 2005, that turned out to be #2, and my bid, the Hans Reiser case, was nowhere close to the top.<br /><br />So, what do Techmeme's sources *really* care about? <span style="font-weight: bold;">Smartphones.</span><br /><br /><a href="http://apple.com/hotnews/openiphoneletter/">"Dear early adopters: Sorry we made iPhone available to the proles. Here's some iTunes. Love, Steve Jobs</a> [smartphones] 152<br /><br /><a href="http://microsoft.com/presspass/press/2008/feb08/02-01CorpNewsPR.mspx">Microsoft's bid for Yahoo!</a> [acquisitions] 138<br /><br /><a href="http://apple.com/hotnews/thoughtsonmusic/">"Dear iTunes customers: No DRM-free music for you. Love, Steve Jobs</a> [IP] 122<br /><a href="http://www.engadget.com/2008/01/15/live-from-macworld-2008-steve-jobs-keynote/"><br />Macworld 2008 Keynote</a> [smartphones, gadgets] 121<br /><br /><a href="http://googleblog.blogspot.com/2007/11/wheres-my-gphone.html">Google announces Android</a> [smartphones] 113<br /><a href="http://blog.digg.com/?p=74"><br />Digg tells DMCA to bug off</a> [IP] 112<br /><br /><a href="http://googleblog.blogspot.com/2008/02/yahoo-and-future-of-internet.html">Google on Microsoft's "hostile" bid for Yahoo!</a> [acquisitions] 111<br /><br /><a href="http://www.google.com/intl/en/press/pressrel/google_youtube.html">Google acquires Youtube</a> [acquisitions] 107<br /><br /><a href="http://engadget.com/2007/09/05/steve-jobs-live-apples-the-beat-goes-on-special-event/">Steve Jobs announces cheaper iPhone</a> [smartphones] 104<br /><br /><a href="http://engadget.com/2007/01/09/live-from-macworld-2007-steve-jobs-keynote/">Macworld 2007</a> (and announcement of the iPhone) [smartphones, gadgets] 101<br /><br />Gadgets, with a smattering of IP and corporate bureaucracy. Spots 11-20 seem to be more of the same.<br /><br />One thing that is worth noting is the "most discussed" story is a single link. So, if a number of news sites "split the vote" and have several discussion links apiece, the story may not surface in this list. With techmeme it is a little more obvious, since most bloggers seem to link to the official corporate press releases. With memeorandum, I'm trusting that preferential attachment (NYTimes and Washington Post dominate) makes it so the vote isn't split often enough to dramatically misrepresent reality.Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-3623899854668090474.post-25824652579223722292008-08-04T13:52:00.001-07:002008-08-04T13:59:38.019-07:00Feed experimentationI'm attempting to redirect my default Blogger RSS feed to <a href="http://feeds.feedburner.com/dataphiles">http://feeds.feedburner.com/dataphiles</a>. Google Reader seems to take awhile to figure these things out, but if you don't see another post from me in the next week, you might check to make sure you're getting the right feed. Of course, given my history there's also some chance I just dropped off the face of the blogosphere, but we'll try to not let that happen.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-3623899854668090474.post-72078816795748913492008-07-31T20:45:00.000-07:002008-08-13T11:23:18.514-07:00Telling us what we already knowVia <a href="http://www.thursdaybram.com/2008/07/25/social-media-the-freelancers-new-best-friend">Thursday Bram</a>, communications agency <a href="http://www.universalmccann.com/">Universal McCann</a> recently conducted the third wave of their global study on social media usage. The results indicated, of course, a growing usage of all kinds of social media worldwide. Also, it notes that "blogs <span style="font-style: italic;">are </span>a mainstream media worldwide and as a collective rival any traditional media" (emphasis mine). Sooner or later, it seems we'll have to be more specific when we say "mainstream media". :-)<br /><br />You can see a complete slide show of results <a href="http://www.slideshare.net/mickstravellin/universal-mccann-international-social-media-research-wave-3">here</a>. <span style="font-size:85%;">(Warning: It's very colorful, and people with a sensitivity to circles should not consume.)</span> It should be useful for citing whenever a convincing intro to a SM research paper is needed.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-3623899854668090474.post-80837567167931253812008-07-31T14:07:00.001-07:002008-08-13T11:23:34.296-07:00What has been happening to the political Usenet?When I first heard about the Netscan project (see <a href="http://research.microsoft.com/%7Emasmith/">Marc Smith's homepage</a>), my thought was, "People still post on Usenet? Last I heard about it, one of the more active groups was <span style="font-size:85%;"><span style="font-family:courier new;">alt.fan.spice-girls</span>.</span>" Working on a related project, I've gotten a similar reaction from other people I've mentioned it to. So an overarching theme of my project has been to answer whether Usenet is a distinct community, or simply a sample of what we already know about online communication.<br /><br />One advantage to studying Usenet is that since it's been around for so long, it's easy to get historical data and say something about its evolution. Furthermore, it's easier to call what we know of it a "community" (although we're still forced to sample it, for our purposes), whereas we never really know if we've crawled*all* the blogs.<br /><br />What we have done so far is obtained data since 2003 for 200 newsgroups with "polit" somewhere in the newsgroup name. Here's some over-time behavior, a plot of number of posts per day, and number of hyperlinks (in original, non-quoted content) per day:<br /><br /><br /><div style="text-align: center;"><span style="font-size:130%;">Posts and Links for All Political Newsgroups</span><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiHFDVNx_aPYUsoGj0RPDUmO13nPc1VYHObqsr4ZcBMKz6TcitCR3v2xpKX_3Tw4AQ6wUE8ARiMOrzPHqqbaa3cdlp5qxRK_Bnd6Ys-lWpqUqCU9mDNzxR0t5cnGM5BFlHK_IrPfiZQJ4M/s1600-h/all-posts-links-over-time.PNG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiHFDVNx_aPYUsoGj0RPDUmO13nPc1VYHObqsr4ZcBMKz6TcitCR3v2xpKX_3Tw4AQ6wUE8ARiMOrzPHqqbaa3cdlp5qxRK_Bnd6Ys-lWpqUqCU9mDNzxR0t5cnGM5BFlHK_IrPfiZQJ4M/s320/all-posts-links-over-time.PNG" alt="" id="BLOGGER_PHOTO_ID_5229330825101446642" border="0" /></a><br /><div style="text-align: left;">This is a smoothed version of the data, so to illustrate a general trend. The first thing you'll notice is the bump in November 2004, which we can attribute to the US Presidential Election. The next thing you'll notice is that while the number of posts is declining, the number of links remains stable.<br /></div></div><br />Here's the same data for a small subset, can.politics:<br /><br /><div style="text-align: center;"><span style="font-size:130%;">Posts and Links for can.politics</span><br /></div><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxoBlWuE9Os9ubqbcail1D-6-S8VhjcArM0xtg2-2hTnuJ-hwUV6iPEbBUJITcmFF0keF5Uptrn6NegHgJV3Cd8eZd9tzYXqTN4fukNwUAC1S52aA44kd05skhyTxUXezZwRBaySwr_wYN/s1600-h/can-politics-posts-links-over-time.PNG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxoBlWuE9Os9ubqbcail1D-6-S8VhjcArM0xtg2-2hTnuJ-hwUV6iPEbBUJITcmFF0keF5Uptrn6NegHgJV3Cd8eZd9tzYXqTN4fukNwUAC1S52aA44kd05skhyTxUXezZwRBaySwr_wYN/s320/can-politics-posts-links-over-time.PNG" alt="" id="BLOGGER_PHOTO_ID_5229332313695679378" border="0" /></a><br />Predictably, the "USA Election bump" doesn't hold for all the groups. For uncultured folks like me who had to look it up, the last <a href="http://en.wikipedia.org/wiki/Elections_in_Canada">election in Canada</a> was January 23, 2006. We do still notice an increasing tendency to link, per post. Perhaps people on Usenet are going more to outside sources. Or, as another intern put it, "They're getting lazy."Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-3623899854668090474.post-34611561335678735682008-07-29T10:25:00.001-07:002008-08-13T11:23:47.804-07:00Scandal sellsSince starting at Live Labs I've gotten to play with a lot of data, including the political Usenet and crawled <a href="http://www.memeorandum.com/">memeorandum</a> hourly data (since mid-September 2005, following Katrina). Today I came across something less-than-surprising.<br /><br />Top 10 links on memeorandum according to most number of 'discussion' links-- that is, number of discussions (usually blogged) that are related to a parent story (usually news).<br /><p class="MsoNormal"><a href="http://nytimes.com/2008/02/21/us/politics/21mccain.html?ex=1361250000&en=33711052dbdd623d&ei=5088&partner=rssnyt&emc=rss">"For McCain, Self-Confidence on Ethics Poses its Own Risk</a>" [McCain and scandals] 219</p><p class="MsoNormal"><o:p></o:p> </p><p class="MsoNormal"><a href="http://nytimes.com/2008/03/10/nyregion/10cnd-spitzer.html?ex=1362888000&en=6ed828c78d717f5b&ei=5088&partner=rssnyt&emc=rss">"Spitzer is Linked to Prostitution Ring"</a> 178</p><p class="MsoNormal"><a href="http://nytimes.com/2007/08/27/washington/27cnd-gonzales.html?ei=5124&en=036c725d4383b4bb&ex=1345953600&adxnnl=1&partner=permalink&exprod=permalink&adxnnlx=1217350840-NUZjHdTDbk7AgSDtrZ64Cg">"Embattled Attorney General Resigns"</a> [Gonzales and scandals] 170</p> <p class="MsoNormal"><a href="http://drudgereport.com/flashos.htm">[Text of Obama's race speech]</a> 158</p><p class="MsoNormal"><a href="http://usatoday.com/news/washington/2006-05-10-nsa_x.htm">"NSA has massive database of Americans' phone calls"</a> 129<br /></p><p class="MsoNormal"><a href="http://tnr.com/politics/story.html?id=8b7675e4-36de-43f5-afdd-2a2cd2b96a24">"The Long Run-Up"</a> [McCain and scandal] 119<br /><o:p></o:p></p> <p class="MsoNormal"><a href="http://rollcall.com/issues/1_1/breakingnews/19763-1.html">"Craig Arrested, Pleads Guilty Following Incident in Airport Restroom"</a> 116</p><p class="MsoNormal"><a href="http://nytimes.com/2006/11/03/world/middleeast/03documents.html?ex=1320210000&en=ba99ceafb0f67900&ei=5090&partner=rssuserland&emc=rss">"US Web Primer Is Said to Reveal a Nuclear Primer" </a>[Iraq and Nukes] 115</p><p class="MsoNormal"><a href="http://michellemalkin.com/2007/11/29/digging-out-the-cnnyoutube-plants-abortion-questioner-is-edwards-supporter/">"Digging Out More CNN/Youtube Plants" </a>[Youtube politics and staged debates] 115<br /></p><p class="MsoNormal"><a href="http://www.commentarymagazine.com/blogs/index.php/podhoretz/1474">"Dark Suspicions About the NIE"</a> [Iran and Nukes] 107<br /><o:p></o:p></p>So, for the most part, what sells is sex and violence.Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-3623899854668090474.post-18451026314221610462008-06-08T11:21:00.001-07:002008-06-08T11:31:40.868-07:00Book: Beyond Fear, by Bruce Schneier<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://www.amazon.com/Beyond-Fear-Bruce-Schneier/dp/0387026207"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer; width: 200px;" src="http://images.amazon.com/images/P/0387026207.01.LZZZZZZZ.jpg" alt="" border="0" /></a>I read this a couple months ago and failed to take it with me to Seattle, so I've lost the notes I took on it, but it at least bears mentioning.<br /><br />He proposes looking at a security problem/solution using the following steps:<br />1. What assets are you trying to protect?<br />2. What are the risks to these assets?<br />3. How does the proposed security solution mitigate those risks?<br />4. What other risks does the solution cause?<br />5. What trade-offs and costs does the solution impose?<br /><br />It's a good introduction to some of the principles and key terms in security (at least, from what I can tell, as someone who knows very little about the field). He uses examples of national security throughout the book, essentially telling readers that terrorism isn't as much of a threat as everyday dangers like heart disease and car accidents, and that the current solutions do not mitigate the risks well. What I liked most about it was that he can frame anything in terms of a security problem and explore it in-depth (including a lot of things I wouldn't normally have thought of in that way, such as maintaining a population of honeybees), which puts it in the category of "books that help you learn to think differently". If I were put in the position to teach an undergrad-level course on computer security I would make it required reading in the first couple weeks, just to get students in the right frame of mind to think about security problems and solutions.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-3623899854668090474.post-15610901263020058442008-06-03T14:02:00.000-07:002008-06-03T22:58:27.073-07:00E coli: not just for health scares<div>Today MSR had <a href="http://www.carlzimmer.com/">Carl Zimmer</a> visiting to give a talk on his latest book <a href="http://www.amazon.com/Microcosm-coli-New-Science-Life/dp/037542430X/ref=pd_bbs_sr_2?ie=UTF8&s=books&qid=1207435111&sr=8-2">Microcosm: E coli and the New Science of Life</a>, following a pre-talk backyard burger grilling (not really). I watched over the live-streaming video. Zimmer addressed how E coli has been used in the past for scientific experiments, and some new directions that microbiology is taking.<br /><br />E coli has been used in bioengineering to make synthetic insulin, jet fuel, and cancer treatments, to name a few. Some students even found a way to make it <a href="http://www.astrobio.net/news/modules.php?op=modload&name=News&file=article&sid=1786">"take pictures"</a>. E coli has around 2,000 "core" genes, while the entire genome (all strains of E coli) has nearly 10,000 that have been found so far (for comparison, humans have 30,000). Some scientists believe that the "bare minimum" of genes necessary for its survival is around 200. Venter and company have already been working with a different smaller-genomed species, and "keep knocking out genes, to see if it still lives." Their count is down to 350. Potential experiments are to take these O(100) genes and begin adding more to create "new life" specialized for some purposes, which is very futuristic-sounding.<br /><br />Other interesting experiments involve finding bacteria that are <span style="font-style: italic;">already</span> suited for human needs. For instance, a teenager in Canada already <a href="http://news.therecord.com/article/354044">isolated bacteria that eat plastic bags</a>. These sorts of experiments could solve a lot of problems. I wonder if there are bacteria that turn lead into gold. :-)<br /></div>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-3623899854668090474.post-82437827354379947302008-06-01T17:05:00.000-07:002008-06-01T18:11:17.117-07:00Newsflash: Flying is FrustratingVia <a href="http://consumerist.com/tag/travel/?i=5011894&t=poll-are-you-fed-up-with-flying">The Consumerist</a>, <a href="http://www.bizjournals.com/austin/stories/2008/05/26/daily27.html">Americans are flying less</a> because it's such a frustrating process, according to the Travel Industry Association. Detailed survey results are <a href="http://www.tia.org/resources/Public_Affairs/ME_KeyPoints_3.pdf">here (PDF)</a>. <br /><br />Oddly enough they don't say anything about fuel costs, which I imagine has a much larger impact. For one, <a href="http://www.marginalrevolution.com/marginalrevolution/2008/05/americans-drivi.html">people are also driving less</a>, and presumably this is not a reaction to the fact they're just sick and tired of having to fight their neighbor for the armrest.<br /><br />For two, people have a greater tendency to grin and bear it when they're paying less for something (just ask any Southwest Airlines customer*). But when flights start costing more, whether on the ticket or by new-and-improved fees ("Now you want <a href="http://seattletimes.nwsource.com/html/nationworld/2004430827_americanbaggage22.html">$15</a> to lose my bag, a service that used to be free?"), people expect a better experience, even if logically they know the cash is just getting pumped into the fuel tanks.<br /><br />Perhaps I'm missing something. I haven't paid much attention to flight prices over the past year; I'm just guessing they've increased. (And if they haven't, that might explain why airlines can't get their stuff together enough to satisfy their customers.) Does anybody have solid data on this? Better, does anybody have solid data on how many people actually fly, not just what a consumer survey says?<br /><br />*- I kid, but SWA flight attendants have been known to say during the pre-flight recitation, "Please do not tamper with the lavatory smoke detectors, as the penalty for disabling a smoke detector is up to $2000. And we know that if you had $2000, you'd be flying American."Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-3623899854668090474.post-75789082812333671202008-06-01T15:19:00.000-07:002008-06-01T16:02:55.374-07:00Started at MSR/LLI'm in Bellevue, WA now, and just finished my first week as an intern at Microsoft Live Labs. I'm working with <a href="http://datamining.typepad.com">Matt Hurst</a> on some social media stuff. So far MSFT has been a fun place to work; everyone seems really happy.<br /><br />One of the things I'm most excited about is the puzzle culture. I did <a href="http://cmupuzzlequest.com/index_old.html">PuzzleQuest</a>, sponsored by MSFT, once awhile back and really enjoyed it. I hear there is an intern puzzle day as well as weekend-long <a href="http://www.interngame.com/">"The Game"</a> (not to be confused with <a href="http://www.losethegame.com/">The Game</a> that I just lost). The latter is apparently invite-only, so I will have to get more details later.<br /><br />Other notes:<br /><br />-We found out that some recent work with Leman and Christos was accepted to KDD, so I will be in Las Vegas at the end of August. With my trusty free Microsoft Research nalgene bottle, so as not to dehydrate.<br />-As I tend to do when I travel, I've done an unusual (for me) amount of non-work-related reading in the past couple months. Will update later with some notes.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-3623899854668090474.post-70509281813017780012008-04-17T14:07:00.001-07:002008-04-17T14:39:37.960-07:00How to make time for literature reviewAnswer: just wait until you're completely unmotivated to do anything else. Sunny days with perfect weather are really the only times I get a chance to do any significant literature reviews. This afternoon, when I was unable to get myself to stay in my windowless office, I (finally) sifted through the WSDM proceedings that I'm most interested in, and read a couple papers on trust/distrust propagation. I'm getting better at adding papers to my <a href="http://www.bibsonomy.org/publrssN/user/username?items=1000">bibsonomy [rss]</a>. The top 10 or so should be what I covered today.<br /><br />Also a fun article: via <a href="http://arxivblog.com/?p=367">Physics Arxiv Blog</a>, <a href="http://arxiv.org/abs/0804.2202">To How Many Politicians Should Government Be Left?</a> The article looks at the "efficacy" of a government compared to its cabinet size, and makes a rather nifty model of how opinions are formed in small networks. Another interesting bit is that while cabinet size ranged from 5 to 54, not a single government of the nearly 200 surveyed had a cabinet of size 8-- apparently it is common knowledge that that is bad luck, or something.<br /><br />I also discovered that <a href="http://www.cs.cmu.edu/~jure">Jure</a> was smart enough to submit last year's SDM <a href="http://www.siam.org/proceedings/datamining/2007/dm07_060Leskovec.pdf">paper</a> to <a href="http://arxiv.org/abs/0704.2803">ArXiv</a>, which yielded a citation. That has prompted me to register so I can post other publications.<br /><br />This is related to a recent pet peeve of mine-- the fact that it's difficult to get conference proceedings. The ACM/Citeseer folks don't always things from workshops and the like that I'm interested in. Most authors have the sense to post their papers on their websites, but I much prefer being able to get a conference all in one place. Of course, professional organizations don't like to do that. I find it hard to believe that they really make money off of conference proceedings, so I can only guess that it has to do with publisher/copyright/legalities rules outside their control. Maybe someday CC/GPL will be able to wrest away some control.Unknownnoreply@blogger.com9tag:blogger.com,1999:blog-3623899854668090474.post-89421899584620592182008-04-06T16:20:00.000-07:002008-04-06T16:28:52.061-07:00ICWSM, semi-supervised learningReturned from <a href="http://www.icwsm.org/">ICWSM</a>, and was inspired to perhaps start blogging again, but we'll see how long that lasts.<br /><br />The <a href="http://www.icwsm.org/2008/tutorials.shtml">tutorial</a> at ICWSM went well (pdf slides available at that link, ppt available by emailing me). I will be giving it again at <a href="http://www.cs.cornell.edu/conferences/nescai/">NESCAI</a>. There were a lot of great talks and posters at ICWSM; a lot more toward the text/sentiment mining side of things than last year, but still a great variety of concepts.<br /><br />While in Seattle I missed the 10-601 class lectures on semi-supervised learning, and had to prepare a recitation anyway. So as part of that preparation I came across a good <a href="http://pages.cs.wisc.edu/%7Ejerryzhu/research/ssl/semireview.html">survey paper by Xiaojin Zhu</a>. It has an entire section devoted to graph-based methods, some of which I hadn't heard of, so this was useful to me beyond giving me interesting things to talk about in recitation. It might be of use to try some of these algorithms on community detection in networks.Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-3623899854668090474.post-69584500492911870202008-02-19T14:38:00.001-08:002008-02-19T14:45:03.754-08:00Open problems in movie stunt coordinationVia <a href="http://www.fark.com">Fark</a>, stuntman is attempting a <a href="http://www.thesun.co.uk/sol/homepage/news/article817428.ece">24-mile skydive</a>.<br /><br /><blockquote> But Steve, of East London, said: “It’s the last great challenge left on Earth. Obviously it will be dangerous. We’re playing with a lot of unknowns. But it’s my job to assess risk and I don’t believe the problems are insurmountable.”</blockquote>Last great challenge on Earth? Looks like all of us scientists can quit our jobs soon! :-)Unknownnoreply@blogger.com1