Cultural issues in CMS R&D

This page contains some ideas about the way our CMS Computing culture affects our R&D options, and is purely my own opinions based on what I've seen in the last N years. If you see anything that's wrong, please correct me!

Nothing here is rocket-science. The basic premise is that anything that can improve our efficiency as developers can only improve our computing system. We should also consider which people we are taking time from, to try to preserve the time used by our most valued people. With that in mind, here goes...

Component Design

We tend to design components in isolation, and implement them in isolation too. So a sub-component of a project is designed within the context of that project, but other projects don't look over to see what's going on etc. This means we can miss opportunities for synergies, taking advantage of bits of code or architectures that have been tried and tested in one context for use in another. We would benefit from having people spend more time looking at what other projects are doing, so we can look for those opportunities and not miss them.

That may lead to a re-factoring of existing components so they can be better shared, or just to the re-use of an architecture. Either way, it's a win.

Similarly, we often develop components before considering how they should be deployed and operated. We should be considering how a component will be deployed and operated from the start, before the code is written. How do we upgrade it and roll out a new version, what downtime or people would be involved in that process, how automatic can it be, how risky or safe is it etc?

Standardisation

We have, for example, several web-interfaces run on cmsweb. None of them have been developed in concert with others, there has been no attempt to re-use standard tools from one to the other. We have YUI, jQuery, home-made CSS grids, CouchDB, MongoDB and probably many more things I don't know about. This has several consequences:

• a developer familiar with one web-interface has to start from scratch when working on another, or is simply not able to contribute to another because the learning-curve is too high and time is too short
• we are exposed to bugs or security issues in whatever software we have, with little control over it. The more tools we use, the more exposed we are.
• we don't even have any guarantees that we are using the same version of external components when we are using them.
• using home-grown libraries where external libraries could be used can mean we have poorer documentation than otherwise.

None of those are show-stoppers, of course, but we could gain from choosing standard components and building them into our own frameworks wherever possible.

For example, the PhEDEx next-gen website was built with YUI-2 plus a home-grown event-dispatching library and CSS. YUI is a standard library, sure, but it's heavy, and closed. It would have been better to adopt something like jQuery which is better known, and now Bootstrap for the CSS layout, and SASS or LESS for CSS management. There are many other lightweight javascript libraries which could be added for specific functionality, as needed.

Deployment and Integration/regression testing

Deployment of code is still painful. Despite the considerable success of the cmsweb deployment model, it's very expensive in the time of our most valuable developers. Similarly, validating and migrating from one OS flavour to another (SLC5 - SLC6, for example) drags on forever and takes considerable effort. We should be conscious of this from the beginning of our development cycle, specifically aiming to ease these problems.

We also spend far too little effort on integration, despite the long-term benefits. Any good integration- or regression-testing tool will allow much of the tedious work of releasing a new version of anything to be either automated, or at least done by someone other than our most valuable developers.

New developer effort

We seem to be in a phase where new developers are often students who are doing three months of service work to earn their credit with the experiment. Certainly we don't seem to be hiring software developers anymore, despite having many leave over the last year or two. Students who don't know programming or the types of computing we do are going to spend a lot of time learning their task and very little performing it. Even if they do manage to deliver something, there's no guarantee of the quality of their work. We can mitigate these problems by designing our components to be as small and modular as possible, even if it doesn't seem necessary from an architectural viewpoint. That reduces their learning-space and focusses them on a small problem, with a better chance of success.

Old developer effort

We have ~70 sites around CMS that transfer or process data. That's a considerable body of expertise available among the site administrators who monitor and maintain their sites. Despite that, we rarely get contributions from any site administrator that can be re-used by other sites.

I know some sites won't consider doing anything that doesn't explicitly benefit themselves, there's no willingness to cooperate and share code or solutions because it's seen as pure overhead. Others are different, but they're in the minority. We all lose by that attitude.

-- TonyWildish - 26 May 2014