perl -lne ‘map{$c{$_}++}(/(\w+)/g);END{map{print}(sort keys%c)}’ **/*.txt

Some additional background on the original problem: the task was stated basically as “in your job as a developer, you’re given a one-off task one day where you need to produce a report of the word frequencies in these newsgroup files, write some code in whatever language you want.” So my intentions with this post were never to say “language X is better than language Y” and my intention for including timing data was never to prove “language X is faster than language Y”; I just wanted to post my particular solutions to the problem and hopefully encourage others to provide constructive criticism on my solutions and provide solutions of their own in other languages. Studying other developers’ code is a great way to learn, and that’s why I made this post.

I learned a lot from viewing the code submitted in comments to this post as well as to the other follow-up posts, and revised my original code:
Ruby: http://gist.github.com/268579
Scala: http://gist.github.com/268580

I decided to remove the timing data because I don’t want this to become a performance test. I also removed the “status update” lines, since we now know these scripts execute fast enough to make that code unnecessary.

The Ruby version now uses Dir[] as it should have before instead of the ridiculous directory-traversal method; I definitely rushed too fast on this part before and should have looked for something like Dir[] in Ruby core.

In the Scala version I condensed the use of the files method into one line, used underscore to represent the file provided to the first block, but couldn’t seem to get underscore to work for word in 2nd block, not sure why.

Regarding the Scala LoC count: only the first 15 lines are problem-specific, the remainder are 3 methods that probably should be in the core Scala library, and probably are in various 3rd party libs. I think those 3 methods show a great power of Scala though, to extend the language so the main problem-specific code is more readable:
– files method just lets you say “do this to each file under this directory”: files(dir) { file => //use file }
– file2String lets you use a File as a String: file.split(regex) <== split is a method of class String, not File
– using is a new control structure that safely closes any object with a close() method: using(dbConnection) {…}, using(reader) {…}, using(writer) {…}, etc.

Again, thanks to everyone for the comments & follow-ups, and let's continue to learn from one another!

http://re-factor.blogspot.com/2009/12/counting-word-frequencies.html

// cat individual files into all.txt

and do the rest in awk one liner

awk ‘{for(i=1;i freqsorted.txt

http://fsharpnews.blogspot.com/2009/12/zach-cox-word-count-challenge.html