TSSJS 2008 Day Three - Synopsis

Session II – eBay Market Place Architecture – Randy Shoup

What happened to Session I ‘s coverage? Lets just blame it on the night before :-)

We all know eBay doesn’t do transactions. Well at least no client side transactions. Their databases run on auto commit. One way they deal with it is by carefully ordering database transactions (example: inserting the slave record and then inserting the master to ensure a consistent master). They also have reconciliary jobs that go through the database and cleanse it periodically.

Strategies for scalability used by eBay
1. Partition
2. Asynchrony
3. Automate Everything
4: Remember Everything Fails

1. Partition

Obviously, they don’t use sessions and they don’t cache business objects (surprisingly). As expected, they use URL rewriting and cookies to track the user. If the data they have to keep about the user is larger than will fit in these two schemes, they use a scratch database. Since they don’t cache business / user related data, they do hammer the databases for all their queries. To handle this situation, they use a custom sharding solution over their ORM and partition their database based on functional divides in the application.

Search: Search queries come to an aggregator which Is actually a scatter-gather (from Enterprise Integration Patterns). This component forwards the search requests to individual nodes which are responsible for indexing and searching just a part of the entire data space. And, then return the results to the aggregator which aggregates (da !!!) and displays them.

2. Asynchrony

The really hard part of massaging systems is guaranteeing once only delivery. If you loosen this restriction, it is a lot easier to scale. They deal with duplicate events by modeling event processing to be idempotent. They deal with out of ordering by making the consumer go to a service that returns the latest state of the event once the consumer receives the event.

3. Automate Everything

Part of that is adaptive configuration: The consumers that dynamically adjusts to meet the SLA by changing parameters like event polling size , number of threads etc. The adaptive configuration also adapts to changes in number of consumer instances.

He gives an example of an adaptive search experience. They have a feedback loop and in an offline way, they analyze it , create a metadata out of it and feed it to the system that uses it to change it’s behavior. Perturbation is the idea that 90% of the time they recommend the optimal. 10% of the time they recommend new options B, C, D etc.. So that if D becomes popular, it will become the dominant recommended value. They also overweigh the negative feedback so that the oscillations are dampened. Pretty slick.

Strategy 4: Remember Everything Fails

Some of the failure patterns used are failure detection, rollback and graceful degradation. Applications log to a message bus and they have listeners that automate the failure detection. It also allows them to detect historical data and it is used from a capacity planning perspective. They get about 1.5TB of log messages every day :-) grep that.

Code rollout/rollback: They have a policy. NO changes to the site that cannot be undone. Each feature has a rollout plan. And, there is a monster rollout plan for the 2 weeks. There is an automated tool that rolls out the dependencies in the reverse dependencies. The automated tool also does rollbacks.

Here is a cool feature. Every feature has a on and off state. It allows them to turn features off rather than redeploying code that lacks that feature. This allows them to deploy features off and then start them later. They are decoupling code deployment from feature deployment. From a developer perspective, they check for feature availability. To blow my own horn a bit, I have built features in the past which can be turned on and off at runtime. I know what you’re thinking (don’t freak out, I don’t) . This is similar to OSGi. Nope not quite. OSGi is about deploying services and controlling their existence. I would say this may be similar from an implementation perspective, but the intent is quite different.

When the resource fails, and it is not critical, it is safely ignored. If a critical service fails, they go to an async mode (and do the processing later) or do failover. When a service does come back up, you don’t want all clients hitting it at once. They have a phased way of letting clients to hit it.

Overall, this talk was very informative. Randy went through a lot of concepts in great detail at a very high pace. I felt there was so much more information left that the talk could have gone for one more hour at the same pace.

Session III – The Busy Java Developer’s Guide to Scala – Ted Neward

A pure functional language has no side effects. But you knew that already. This talk focused on giving an Scala intro to a Java developer. Again, I am not going to cover much of this talk as you can read about Scala yourself.

Lunch Keynote Panel: Patrick Linskey, Ted Neward and others with Eugnee Ciurana as the mike boy ;-)

The conversation went towards the over abundance of frameworks in java. One good point made was, you should never sit to write a framework. You build an application and then extract a framework. In that case, YAGNI will be inherent in that effort. One of the very few web frameworks that was built like that is Rails.

Another point is, it has to be usable before it is reusable. Good one.

In answer to whether the appearance of free type(terse) languages (where syntax does not matter much) java will allow syntax to be optional, it was pointed out that if you try to shoe horn additional features into the language, it will fail under it’s own weight. The main thing in Java is the platform and the APIs and frameworks that are available. Additional languages that run on java platform but support a whole new set of features will see the light.

Session IV: Map Reduce – Why does it Matter – Eugene Ciurana

We looked at Map Reduce and worked out through implementation scenarios in various business domains and the problems or hurdles that we would face in arriving at the solutions using Map Reduce. Audience got to participate and overall, it was a good exercise. (There, I can talk in bullshit business lingo too :-) )

I'm sure you've seen this already, but it is too cool not to point out
http://members.aol.com/matt999h/bullshit.htm

TSSJS 2008 Day Two - Synopsis

Session I - Concurrency: Past and Present – Brian Goetz

I have heard Brian’s talks in the past in the No Fluff Just Stuff conferences, and they have always been enlightening.

Brian recommends these papers about concurrency:
coping with parallelism treiber 1986
why threads are a bad idea, osterhout 1995
the problem with threads, edward lee – 2006

The talk is a bit more targeted at junior developers than I had hoped. Brian dealt with deadlocks and deadlock avoiding mechanisms. Brian talked about STM (Software Transactional Memory) and why he thinks it is not the silver bullet. He does not mention the reasons. He says he is thinking of presenting them in a talk in JavaOne.

Session II- Performance Puzzlers: Kirk Pepperdine & Brian Goetz

What followed were general commandments of performance (at least at this day and age). Release memory variables soon. Perofrm benchmarks on a dedicated machine.

One interesting point that was made was that in a benchmark with a system using OR mapper and a similar one that didn’t the one with the OR mapper performed better. The reason was that the concurrent garbage collector was essentially stealing processor cycles to clean up all the objects created by the OR mapper and ended up throttling the system against flooding the database with requests. When the OR mapper was missing, the database was thrashing leading to worser performance. So, even though both the direct JDBC implementation and the ORM implementation hit the database with similar number of requests, the ORM implementation is not flooding the database and is providing throttling for the requests. Interesting, but this is one of the things that happens by accident.

Session III- Implementing an ESB solution using Mule – Ross Mason

Services can be categorized as being in one of three layers.
Task based – represents a business task – example: buy a product – little reuse
Entity Based – represents a sub task – example: bill a customer – some reuse
Utility based – represents an atomic independent service – example: credit card processing – most reuse

Reusability of a service usually depends on which layer the service is classified into. With most reuse coming form the lowest. (as expected)

Mule provides support for creating functional test cases which, I think are more useful than testing one single class in isolation. More about unit testing vs functional testing in a later post.

Mule Expression framework is a way to evaluate expressions on a mule message. This makes content based routing easier. The expressions can be in xpath, groovy and a whole lot of other types.

Lunch Keynote: Why the Next Five Years will be about languages – Ted Neward

Ted built a case of language oriented programming. Object oriented programming is not the pinnacle in the evolution of languages. I seriously doubt if we will ever reach the pinnacle :-). One of the required features of a language will be tool support. These days it is a lot easier to get tool support for new languages. And, many of these languages can be made to run on the same platform that we currently run on. In the near future, we will see a lot more languages prop up into our mainstream development. Overall, the keynote was great.

Session IV: You got Your Ruby in my Java – Chris Nelson

Chris gave an introduction of Ruby. This talk is basically a Ruby introduction to the Java developer. I am not going to list the content of the talk as you can google it up and read about Ruby. It is nice that we as an industry are finally breaking the language stalemate that we hit and are willing to look at other languages to improve productivity.


Fire Side Chat: Concurrent Programming with Java and Erlang:

This was the first fireside chat that I’ve been to. The closest I’ve been to a fire side chat is viewing a chat on Google video :-). This free form of discussion with the right participants can really touch on various topics. Something like the discussion on TSS without the flame wars :-).

Fire Side Chat: Mission Critical deployment at Leap Frog Systems - Eugene Ciruana

I have to admit that I don't exactly remember the name of the presentation. But, this is by far the best one I've attended at TSSJS. The crown jewel if you will. To give an analogy, if you've read Patterns of Enterprise Application Architecture - Martin Fowler, it is the embodiment of years (if not decades) of rich software development expertise condensed in a book. This talk was the embodiment of years of practical, i repeat, practical software development experience condensed in a 1 hr talk. This was a fire side chat and not a typical presentation. We the audience, got to ask all sorts of questions about Leap Frog's infrastructure and more importantly, the design decisions that went into selecting the software solutions. It was like discussing the battle plans and reviewing the war with a war general.

No matter how many subject matter experts you talk to, it is the experience in the field and in the trenches that really teaches you and really counts.

TSSJS 2008 Day One - Synopsis

Key Note: Neal Ford, Thoughtworks

Neal started with an assault on Ruby of Rails developers(and later justified the reason why ruby on rails developers brag). I think the reason why they brag so much is because it is so productive. Thankfully, on the java side, we have Grails to the rescue. Anybody who hasn’t taken a look at Grails absolutely should.

Next, he builds a case for DSLs. One of the strong points of DSLs is that the context is implicit. You don’t have to keep reminding the runtime of the context:

Example: Consider the following java code

Car car = new Car();
car.setColor(Color.BLACK);
car.setTransmission(Transmission.MANUAL);
car.setPrice(30000);

Now take a look at this in a dsl that I just made up

create Car:car
color:black
transmission:manual
price:30000

Now that is much more clearer as we don’t have to keep ourselves repeating the context which in this case is car.

One thing you have to give to Neal Ford is, he knows how to give presentations. In his presentation, he normally uses just one sentence per slide, Sometimes just one word. And lots of pictures. These pictures are typically not from the computer science domain. These are usually analogies from the real world that help the listener in understanding the underlying concepts. Another technique I noticed him use is when he wants the user to concentrate on his talk and not on the screen, he leaves the screen blank.

Also, a template for your presentation is not needed. Leave it dark. It works magic with the audience. Templates and light colored background distracts the listener from what you want to show. This style of presentation is also visible in Steve Job’s presentation. He never uses a template.

Ok that was an un-intended tangent. Now back to the talk: As a programmer, it adds real value to write programs that are close to how the user talks about the problem. It is a huge advantage if the users can read what the programmer can produce..

The final point was that the development stack of the future would be a structured programming language at the bottom. A dynamic programming language on top of it to improve development productivity and then a DSL on top to stay close to the problem space.

Tactical Design by Glenn Vanderburg:

Presentation tip: Try to build on key aspects of prior presentations (the ones you like) so that the audience get a transitionary feel.

His talk concentrated on how to improve the design skills of poor designers and make them good designers. I think it is obvious that DRY and sticking to one level of abstraction thought the method is a good way to improve design. This is analogous to saying “use Design Patterns”. It is also ironic that he took the same topics and repeated them over and over. Overall, the talk was sub mediocre. Luckily, I had my laptop with me and caught up on some grails reference documentation.

p.s: Lenovo sucks. But having a built in upgraded battery back that gives you 6 hours of alive time rocks.

Also, if ure in a talk that you are not sure of, sit at the last so that you can sneak out if the talk sucks. (Also iterated by Hani in one of the previous years of TSSJS)

Self Scaling Java Based Cloud Architecures – Jinesh Varia, Evangalist at Amazon

I can say I hate talks given by evangelists as most of them tend to be biased. The talk sounded like it was a product presentation and less about giving insight. It takes a visionary to be able to give insight. I thought I can stick around as I wanted to know more about Amazon Web Services. Basically it was a vendor talk, and does not delve into the methodology or mindset that leads to cloud computing.

After 30 minutes, I could not bear it anymore and snuck out.

Next stop: Building REST-ful web services with the JAX-RS API – Mark Hansen

This talk turned into a ‘show and tell’ and I snuck out as soon as I typed the title.

Speed: Kirk Pepperdine

This was the third talk I went to in an hour. You can’t go wrong with a talk about JVM J. The talk covered improvements in garbage collection times and algorithms. One of the cool things that Java 7 (sun jvm) will do is look for local references and allocate them on the register. So that they never see RAM.

After the talk, I caught up to Kirk and asked him a few questions about when you would do volatile (for primitives) vs when you would do AtomicBoolean(and other atomic classes). More on this on a separate blog.

Designing for scalability: Patrick Linskey

This talk was about horizontal vs vertical scalability. Running into contention because of shared state. And, about partitioning the application at various levels to achieve scalability.

An application can be partitioned in multiple ways. Some of which include:

Partitioning along application bottlenecks.

Patritioning along data set “fault lines”. Example: geographic collocation so that only relevant information is close to where it is being used. This is an example of a stateful service being partitioned for scalability.

Also using asynchronous execution to increase scalability. Nothing ground breaking. One good point made was that asynchronous architectures inherently support throttling so that the system can catch up on processing in off peak hours.


Boldly Go where the java language has never gone before – Geert Bevin

This is an interesting frame of mind that you do not need to leave the java language. The language can be instrumented or transformed into a different execution platform so that the developer skills can be reused. Geert gives example of Terracota which performs dynamic byte code manipulation of classes to provide clustering without the developer having to use a vendor specific API.

The next example Geert touches is GWT. GWT lets the developer code in java but generates javascript under the covers in production. I would think the intent of GWT is nice. Free the developer from having to worry about javascript incompatibilities between different browsers. The implementation is based on java, which I increasingly think is a wrong tool to create web based/GUI applications. The Java syntax is too buttoned up. Grails on the other hand is a wonderful tool to create web application. I think a grails implementation of GWT would really shine.

The next example is Android. This is my favorite. Android, as you might already know lets the developer write code in java, but gets compiled into an executable that gets interpreted by the Dalvik virtual machine on the cell phone. Although the developer is coding in java, like in GWT, what gets executed has nothing to do with the JVM. I have not done any serious Android development. My take is that UI development is better left to templates. I think it is just a matter of time that a UI templating framework is released on top of Android.

The scalability pitfalls of the realtime web- Jonas Jacobi & John Fallows

The presenters build a case for a event driven server which is basically a reverse ajax. And started listing the pitfalls and solutions(or hacks depending on your perspective) to these pitfalls. Snoozfest. And yes, I got out as I could.

There was only one other presentation that was remotely interesting, which was a use case on SCA.

Next Generation Payment Systems using SCA

My take on SCA is that it is trying to make the same mistake that EJBs did with the “thy shalt give thy java interface to the client”. I am a big proponent of simple and non language specific contracts between multiple parties in a distributed platform. When you expose a webservice (even REST based ones) you are essentially creating a contract with the end developer about the nature of the data interchange or service invocation. The problem with a language specific contract is that apart from being tied to the language, you now get into headaches with versioning. Most programming languages that we currently have do not work well with versioning. So much so, that these days, we have had to build specifications like OSGi to handle multiple versions of an object existing in the same virtual machine.