Losing the Big Picture

Sometimes it happens that we tend to get lost in following some
processes/standards so religiously and lose the sight of big picture or
the essence/crux of what we intended to achieve in first place in a
timely fashion.

Following processes/standards is good but it should not hamper or come
as a stumbling block to achieve your principal objectives in time when
they matter the most.

Anyway this is not something unique to Software Industry but in general
applies to any walk of life. From time immemorial all religions
themselves have seen many new philosophies emerge inside their realm
whenever people following the existing ones lose the sight of big
picture and become too much involved in processes to the point of losing
the essence/crux.

Man is a tool-loving animal

Excessive proliferation of tools and technologies in today's software
(this is also good in some aspects,thanks to open and/or free software
movement) ecosystem illustrates this and the plethora of seemingly
similar tools/technologies in IT can confuse even the best learned.
I often find too much engineering where there would hardly be any
significant difference in functionality and/or performance (Scaling is a
different phenomenon with different degrees of expectations and
compromises).No wonder we as humans are tool loving animals and also the
fact that no tool is even near perfect to cater/accomodate to all
situations at any given point drives this need.

5 Tools that i am personally interested in field of Application
runtimes/profiling are:

i) DTrace on Solaris ( may take some time for me to try out on linux)
ii) JRockit Flight Recorder and any plans from JRockit team to
supply a friction-free logging library ( ps: i am not starting any
raging controversy here on whether logging is good/bad or
merits/demerits over diagnostics)
iii) Azul systems diagnosis
iv) JXInsight
v) Yourkit

( in no specific order pls !!!!)

I havent used/seen DTrace for java apps so i do not know at this stage
how it could help to lead all the way up to showing stack trace in
JVM/user space esp in context of JRockit where the
code-compilation/conversion is different than Sun Hotspot JVM which
belonged to Solaris land.

Back to sql land, i had always maintained model is the code in all
walks of software and in particular Dan Tow's diagrammatic way of
visualizing a SQL to understand if CBO really chose the best possible
plan ( one can argue it is the job of CBO and why we bother the internal
algorithm of choosing a plan etc and trust me you would need it at some
point!) had been running in my mind quite for sometime since 2006.
Even wondered why someone didnt take it up to automate that to provide a
visual way of looking at things and stumbled upon this article
(http://sites.google.com/site/embtdbo/sql-tuning-1#TOC-Visual-SQL-Tuning-in-Action
) which shows that people do have similar intentions as mine. In
Software esp in performance management there is no revolution but a
constant evolution of ideas and thoughts that drive things.

Converging JVMs and DTrace for Linux

The news is officially out and as expected the 2 popular JVMs(Sun
Hotspot and JRockit) are getting converged and also a DTrace port for
linux getting started to mature.
The JVM itself being a C/C++ runtime would in my opinion go through some
changes esp in context of better diagnosis and better integration with
other underlying layers in future but improvements in
performance/scalability need to be tested out as it may not be too clear
at this point

Benchmark wars and Number Games

What differentiates a seasoned performance engineer from a
developer(however senior he/she ) is that the seasoned performance
engineer doesnt have to rerun several test runs periodically of more or
less same load nature/workloads to identify bottlenecks in code and can
spot issues in design that can cause performance and scalability issues
in long run much more easily with careful/concentrated fewer test runs
saving lot of heat in arguments(saving fuel and energy) and most
importantly much earlier in dev cycles helping everyone.

Sounds nice to have such folks on board! But wait a minute all this is
good only if the Dev folks listen to the voices and in a predominantly
developer dominated organizations it simply doesnt happen due to various
reasons and even the seasoned performance engineer goes through tough
times of re-running tests again and again to spot the same old
things/cries which he/she would have already cried on top of his/her
voice a thousand times over the roof.

Also much more valid is the quality of ideas and tools he/she uses to
arrive at results quickly can often surprise even the most senior dev
folks that they initially try to initimidate/resist such changes.

No offense meant to any developer folks reading this!

Impatient Patient and a Helpless Doctor Syndrome

Working to help in performance and stability in agile development
environments/developer eco systems poses some interesting challenges for
performance engineering folks.

I get reminded of what i call a "Impatient Patient and Helpless Doctor"
syndrome in such eco/environments.

The development staff mostly very senior and already given you an
impression of they know/understand their code/systems they develop well
enough that they want to be helped only in understanding what they need
or think need to be done.

This is like a Impatient Patient but very learned/educated at same time
requesting the doctor to get him/her checked and get rid of all "XYZ"
scalability and performance issues. What follows is the doctor initially
goes into a Helpless mode as the Patient itself suggests all
methods/tests be performed on him/her and often indicative of the
cure/medicine for the assumed illness.

Not to say that only a Successful doctor helps the patient to come out
of such mental concoctions and manages to truly help the patient in the
end winning their confidence. But this doesnt come that easily and
chances that the doctor though clever enough can become crippled and
helpless to the extent of losing interest in the subject can very well
happen.

Lesson learnt is : The patient should allow the doctor to do his/her job
without imposing too many blockades and doctor also give some mental
peace to the mentally agitated patient along with cures. Needless to say that the patient need to be cured really but providing mental peace to patient helps.