The churn of being cutting edge

Stream processing is a hot topic these days, and I've had to learn quite a lot about it at work for the past two months.

There is an endless number of frameworks for stream processing. To name a few,

• Samza
• Storm
• Spark Streaming
• Onyx
• Kafka Streams

Having choice is not necessarily a bad thing, but having too many choices has downsides like analysis paralysis, changing horses midstream, and incompatibilities between different platforms.

Choosing Kafka Streams over Samza, for example, is a good decision if you're using Kafka as your messaging system–but the cost of porting your application(s) is non-trivial, and can significantly add to the project's timeline.

The rapid development of technology has another undesirable property: rapid deprecation.

With so many technologies–and companies behind the technologies–in competition, no one can afford to provide support for older versions of the technologies. And since development is so rapid, "old" can mean as little as 12-18 months.

This imposes a large burden on users of these technologies to upgrade their software instead of focusing on the business needs, which will, again, add to a project's timeline.

Don't think upgrading versions is a trivial thing–version dependencies can get real complex, real fast, and an attempt to upgrade one library in one app could end up requiring upgrades to many libraries and many apps. This is especially painful for a small team, which lacks the manpower to constantly maintain libraries.

So is it worthwhile to be on the cutting edge?

For businesses whose primary concern is not technology–that is, the business is primary a consumer of, not producer of, technology–I would say that it's good idea to wait a while for technology to mature.

After all, if you're not selling tech, there's no business interest in getting to the forefront of it. It's good not to lag behind too much, as many business are wont to do, but it's also wise to avoid being a trailblazer and encountering all the difficulties that entails.