Go back to startpage
.. some thoughts .. after many years ..
This website is dedicated to K/Q/Kdb+. I have been a professional K/K4/Q/Kdb(+) developer for over 25 years.
I volunteered in testing K4, the scripting language of Kdb+. Before delving into the K language, I spent
nearly 11 years developing in Sharp APL (and J) at the good old SKA (Schweizerische Kreditanstalt) in Zurich,
Switzerland, which was renamed Credit Suisse Group AG in 1997. I left CS in 2000.
Go back to startpage
... but lets go back to the roots ...
The 1970s and 1980s were a technological dawn, marking a revolution. Programmable calculators (like HP's and the
famous Texas Instruments TI-5* series) fascinated me. They made it much easier to tackle laborious numerical
problems and provided room for experimentation (and therefore 'to boldly go where no man has gone before').
For someone with an affinity for mathematics, this timing was a perfect match.
Slide rules and logarithmic tables have become part of history.
My programming experience began with the Texas Instruments calculator TI SR-56, introduced in 1976, which featured 100-step program memory.
Later, I became the proud owner of a TI SR-59.
Punched cards, floppy disks, and later magnetic cards were the state of the art at the time. My first real computer, an Apple IIe,
had a memory of 32 KB (that's not a typo), and the upgrade to 64 KB cost me over 400 Swiss Francs. The Apple IIe was equipped with a
slot for a 640 KB 5.25-inch floppy disk drive, which was so loud that whenever the PC started reading or scanning the disk, my
house pet - a small blue parrot that had flown to me two years earlier - would panic in my student room. Today, the cost of an
M3 MacBook Pro is only 25% to 30% of what I paid for the Apple IIe (when adjusted for the present value of the original Apple IIe price).
Long before joining the bank, my programming experience included classical languages like Pascal, Modula-2, and bits of BASIC.
However, what I liked most as a student was Assembly language; I programmed in Assembly on the MOS-6502 microprocessor and the
Motorola 68000, 68010, and 68040 series. These Motorola chips were excellent. I wrote Pascal and Modula-2 programs with 680xy Assembly subroutines
on an Atari computer, which was not that easy because some important documentation was unavailable for the Atari (at that time, it
was primarily a gaming computer). Despite all the fun with Assembly, it was not an ideal environment for mathematical challenges.
After university - and some traveling and earning money through temporary math teaching jobs to finance my first car -
I joined the SKA in Zürich - Oerlikon around the year 1990. Initially, I had to learn Sharp APL, which came with its own
operating system called 'Logos' (APL was invented by Ken Iverson in the 1960s). I was fortunate to join a small team of great developers,
and I immediately saw that APL was great to apply my mathematical skills. (I will probably never understand people who study anything solely for the
sake of obtaining a title.) APL, as a functional programming language, was absolutely ideal, so I quickly familiarized myself
with the pricing of derivatives. About four years later, the application needed to become real-time, so I built my first real-time 24/7
pricing server, which included my own pricing algorithms that also allowed verification of the traders' portfolio results. This server ran
worldwide within the bank's network. I also developed algorithms to detect trade fraud based on traders'
activity data and was deeply involved in system integrations.
At that time I had the honor of meeting Arthur Whitney in person, the inventor of A+, K, and Q (Kdb+). He demonstrated some
programming examples of K, which would later become a significant part of my daily life as a senior analyst, software architect,
and developer. To me, K symbolized the future - a magical programming language designed to handle complex data
and algorithmic problems, particularly emphasizing its volume and speed.
However, K was unavailable to the public for several years, so I transitioned to J
(while continuing to use Sharp APL). Despite this shift, I am part of the first wave of K(and later Q/Kdb+) programmers in history.
J, invented by Ken Iverson and Roger Hui in the early 1990s, is another beautiful functional programming language.
By the way, J and APL influenced the development of Python's NumPy (Numerical Python extensions) package.
Since the year 2000, I have been using K on a daily basis, and since 2003, I have also been working with Kdb+ (initially on Windows,
and later on macOS and Linux). Typically, I wrote applications that worked on various operating systems.
Over the last ten years, I have also coded in Excel VBA, MS Access VBA, MS Word, SharePoint, Java, R, Python, and some SAS (when I needed
to replace a major SAS application with Kdb+), as well as C++. As a K/Q/Kdb+ developer, I found particular enjoyment in working with
Python due to the natural interface it offers between Q and Python through embedPy and PyQ..
.. Some observations through time ..
To a large extent, my professional life has been focused on building major architectures and applications in banking and eCommerce,
developing complex algorithms was part of the game. In other words, I typically served as an analyst, architect, and developer all in one.
I have always straddled two camps: business and IT.
Business specifications are never set in stone; to be more precise, they are often brittle. Sometimes, crucial information arrives too late
(or even after implementation) or becomes a moving target. Pursuing a complete rewrite should not be confused with unhealthy and pointless
perfectionism - there is no such thing as perfect code. However, repetitive 'optimizing' and 'tuning' often indicate that not everything is
functioning well in the implementation or architecture. Experience tells us that 'luck is the residue of design'.
Over the years, I have encountered various self-delusional 'recommended managerial process strategies' that plagued business departments.
One major misunderstanding is the so-called 'simplification approach.' This is a managerial attempt to split a process into simple pieces with
the hope of hiring cheap rookies to maintain the application, ultimately firing the original staff. It's similar to trying to create a cube out
of two squares - it may look cool and feasible on PowerPoint slides, but it doesn't work in practice.
Instead of investing in a few skilled and motivated individuals who communicate effectively, it seems "safer and cheaper" to address
communication gaps primarily through numerous meetings. These meetings often include participants from diverse backgrounds and skill sets
who may not fully understand each other's roles or whether they even know their own responsibilities, which affects how effectively
they can trust one another to execute tasks. Such situations are often accompanied by micromanagement, which stifles any hope of improvement.
In this context, performing a job one dislikes is like poison for the soul, leading to unhappiness and dissatisfaction for both individuals
and those around them.
The other problem is more psychological. These people, who are supposedly easily replaceable (and therefore cheap), are unfortunately often
treated like commodities, so their motivation is generally not overwhelming.
Furthermore, the more people involved, the higher the likelihood of fluctuations, and handovers almost always lead to some temporary
or permanent loss of information.
The nearly consequential phenomenon is that over time, a few key players might emerge, leading to the exact same situation that was
supposed to be 'avoided.'
Another trap to keep things "simple" is procedural cookbooks. Here a nice example:
(I) Run this SQL on this database
(II) Save the data as csv using tool X
(III) Open the file as Excel and save it as macro enabled
(IV) Replace the header with this new header
(V) Merge column 7 with column 12
(VI) Run macro XY
(VII) Import the file into Ms Access
(VIII) Run Query xyz etc etc etc.
Typically, you discover that such 'cookbook applications to keep things simple' are written by people who left the company a long time ago.
As a result, the whole procedure becomes an iceberg that nobody wants to touch, yet someone needs to manage it because, for some reason
(which nobody can explain), the process is still needed simply because no one wants to take responsibility for shutting it down.
People come and go, and the process passes from one 'dynasty' to the next until a major reorganization takes place. Eventually, the application
fizzles out almost unnoticed, unless something happens elsewhere, leading the responsible parties to be unaware of why changes occurred or
whether they are important..
... outsourcing ...
Outsourcing itself is not necessarily bad or wrong per se. However, outsourcing more complex applications and their functionalities - especially
when they are not understood to a sufficient level - can lead to significant challenges for all parties involved. Complex applications should
typically not be outsourced, and communication channels should be as short and direct as possible. Additionally, the tempting strategy of
outsourcing to 'low-salary countries' can quickly become a mirage; the business and job market are global. To keep salaries low in such countries
will simply motivate the people there to look for better paid jobs elsewhere, meaning salaries can rapidly average out,
regardless of whether your remote team consists of independent contractors or employees of a larger international software company.
... artificial Intelligence ...
AI is undoubtedly very helpful in many aspects of human life, but it should not be portrayed as something 'super-human.' This 'super-human' perception
should be firmly rejected, as it reflects a significant misunderstanding of what AI truly is. AI should definitely not be viewed as a replacement
for human intelligence or any human-related qualities, such as creativity or feelings .. In (pre-)schools, hands-on activities like working with
pencils and paper remain essential for brain development and should never be replaced by IT solutions.
Critical thinkers must consistently scrutinize any proposed answers,
... data quality ...
One heavily underestimated field is data quality. Unfortunately, it seems easy to write numerous lengthy books about it without any hands-on
experience. There is no panacea that 'cures' data quality problems.
... Technology change ...
Quite serious mistakes and misconceptions occur when a larger application is written in software technology 'X' and needs to be
moved to technology 'Y.' Software is not merely distinguished by the way one writes for loops, while loops, do loops, case statements,
or which packages to use. Software inherently involves a philosophy of thinking, particularly regarding how to approach and generalize a
problem. Often neglected factors include language expressivity and memory management. The worst kind of software solution strategies
(happens more often than you think) I have encountered are those where people look up code snippets in the internet and
copy-paste them into their applications.
... Scrum, Agile ...
When it comes to Scrum and Agile project management, I wish it would vanish from Earth like the dinosaurs - preferably with a loud roar and a puff of smoke!