Sunday, July 6, 2008

Scala/Erlang Differences

I recently read this article at infoq and wondered if anyone could shed any more light on the debate?

Having looked at Erlang for a while, it seems like a refreshing approach to the accelerating needs for concurrency, but I confess to neither being confident with Erlang, nor having done much research as yet with Scala. I am not convinced that the fact that Scala is built on the JVM is actually an advantage, but it is a differentiator. Perhaps it will just be a case of paying your money and making a choice.


Brad Wiederholt said...

I want to comment quickly, but do want to come back to the Erlang/Scala discussion too. Couple of quick things I noticed in the article that this article pointed to:

1) IBM has a concurrent language called X10 they are pushing? Never heard of that, I shudder a bit because of recent IBM experiences but I won't let that stop me from looking at it.

2) The 2nd article quotes a 3rd article that brings up the point that once we get hit in the head with the slowdown in acceleration of processor speeds, can the industry continue to sign up for and develop feature rich bloatware (my word)? Are product managers going to slow down their expectations as well or are we going to try to continue to fit more and more into the same-sized or only slightly bigger package?

Paul said...

Hi Brad,

It appears that "the powers that be" may be waking up to the need for simplicity:

IMO we've got ample processing power for most of the things we do with computers now (assuming that the software is designed right ofcourse :)). More sophisticated user interaction could be a way to soak up those extra cpu's. 3D graphics perhaps?

Where do you see the demand for extra processing power comming from?


Andrew said...

Actually Paul poses a very interesting point and one I admit I had not stopped to consider in all this. This infoq articleand this one in Dr Dobbs and others like them refer to the fact that CPU manufacturers are moving increasingly to multi-core architectures because they are reaching the limits for pushing ever higher clock speeds and other strategies with current technology.

If you read the Dr Dobbs article, Herb asserts that demands will certainly grow in the future to a degree that the traditional architectures can no longer provide for. My instant reaction to this was that of course that is correct.

While I am sure this is true for some applications, on reflection however many applications probably won't need more power, just far less complexity. Does your simple CRUD application for 100 users need to be written in a concurrent language? Possibly not. Its all about choosing the right tool for the job. We don't make anywhere near enough of the power available to us today, instead choosing to load CPU's with pointless work in the form of application/portal servers. Keep it simple and there will often be plenty of available resources much of the time.

Having said all this, I still think a very real need will emerge for the reasons Herb mentions, for us to embrace concurrency oriented programming in the big way in the future - like I said earlier, lets be sure though to use the right tool for the job.

Andrew said...

Just had another thought. Paul is actually teaching me a valuable lesson. He is using agile values to question why, to establish if there is a need. Goes to show how easy it is for me to miss the obvious.

Bear with me guys, I keep trying!

Brad Wiederholt said...

Thanks for the links guys, cause it really proves that I am not a couple of years behind in thinking about this, I am *5 years* behind in my thinking. I have a lot of catching up to do.

Putting on my generalist's hat though, I still think there are folks out there who are as ill-informed as I am who have presentation slides that discuss Moore's law and how things are just doubling along like they've been doing for 40 years. Unfortunately, these may be the same people are are pushing a company along a path based on this assumption. Back in my technology forecasting days, we'd call period like this a discontinuity and would need to go back and redo fundamental assumptions and forecasting models. We'd even rethink a lot of the "ecosystem" type economic models. We'd consult a lot of experts in the field who would tell us about the sea change, that things will change, but were bad at predicting what the future was really going to be like. (None of the Information Research experts we interviewed in 1988 were able to predict the WWW and Google, that were just a handful of years away). I am not convinced that all 'powers that be' are accounting for the discontinuity and it'll be business as usual for them in their assumptions about Moore's law and computing technologies. Folks have been lazy and have let doubling in power cover up some of these mistakes and need to now start thinking about being more parsimonious in implementation.

So what are the application opportunities to take advantage of multicore? Machine intelligence, vision, databases, signal processing, image processing, graphics, simulations, medical research, networking. Yep, these aren't your typical CRUD systems. But IT-like stuff was never a good candidate for parallel processing. I sort of cringe when I think of the shrinking supply of good software developers going into and coming out of schools. I also cringe when I think how hard it's been to get some people to use simple frameworks like Struts, and now those folks will slowly migrate over to the types of applications above...

You guys need to realize my state of mind at the moment: I've created a lot of FUD in my head for myself, and it's going to take me some time to sort it all out. Please temper my comments at the moment as those of an ill-tempered curmudgeon who's painting with a broad brush. Luckily this curmudgeon is open to learning and change...

Paul said...

Hi Brad,

Partial thoughts are good.. Its a sign you're thinking :)

Not wanting to side swipe the debate, but there is an intersting blog comment from Alan Kay question Education and our abilities to think knowadays:

My question was genuine and not a statement. I think you identify some good examples. I come from a telecoms background, and for real-time telecoms systems then something like Erlang is exactly what you need. In fact I've implemented the Actor model myself using nothing more than a realt-time executive and C.

I agree that IT has been the tail wagging the dog in Software Engineering for a while now, which is why I guess that Alan Kay is so upset about the "vocational training" in our Universites that currently passes for and "Education". IT is not a demanding application domain, and I'm not sure that it ever will be.

So for IT CRUD, the performance bottle kneck is I/O and not the cpu, which is why Ruby (on Rails) can get away with running like a dog. Most of the time the cpu is sat there doing nothing. So distributed caches and database technologies (shards etc) is where its at here.

Having said this, if the hardware guys build it, then tthe presence of mulit-cores inthemselves will generate demand I think. What will this mean for the Web/IT world?

The area that can improve the most is user intraction I believe. So soemting like "The Matrix" or the "holo-deck" would be great, a 3D immersive world as the ultimate expression of a "collaborative" world wide web experience.

People are working oh this now. The performance bottleneck here is 3D graphics, which has been overcome in recent years through hardware graphics accelerators. Croquet uses a Squeak VM with primitives that callout to an OpenGL graphics library written in C++. This then relies on dedicated hardware. So 3D performance as a concern has been abstracted away from the programmer.

Having worked on real-time systems, I don't believe that the average IT organisation could deal with the demands presented by live real time data. They aren't that organised. Perhaps what could help here is some type of artificial intelligence. Fuzzy logic that mines data, makes inferences and present strategic options to senior management. Perhaps agents that trawl databases and the web looking for business opportunities?

My partial thoughts added to the pot:) I agree with you we need to be thinking about these things and prepared for all eventualities.


Paul said...

Brad you could be right.

I found this post on artima

Don't look now, but the Thinking Machines approach (op cit) is closer than it appears in the mirror. See, for example, Texas Memory Systems. See also DB2 and Oracle, both putting more/much effort into memory based databases (through acquisition, of course). Relational databases are inherently parallel algorithmically (think: set theory), and take to such machines as ducks to water. The only difference is that now such machines will be available to smaller organizations.

The multi-core/processor solid state machine will be a disruptive event in the business/database side; less so in desktop.

If databases move off the disk into memory, then its a whole new ball game. The question then becomes can the CPU keep up with data access?

I'm not wasting time. I've got going with Erlang already :)