Overfitting – The most important problem we don’t consider enough!

In statistics the name given to mistaking noise for signal is known as “overfitting”. When delivering new Information Systems the concept is also prevalent, particularly at the stage in delivery that sends shivers down every IS project managers spine, the requirements stage! 

Trying to “overfit” evolving business needs into a clearly defined and agreed set of system requirements is one of the most common causes of barriers to business acceptance of new systems. Persuading a business that the requirements have now been baselined and what has been agreed is now what is going to be delivered is quickly becoming a key skill of an IS project manager. Guarding the requirements to avoid any scope creep whilst maintaining business engagement is quite possibly the Holy Grail for IS delivery success. 

My organisation is a few short months away from delivering the single biggest system change in its history, and at the same time as a significant organisational restructure and a focus on the delivery culture of the organisation, guess which element of this huge change programme is considered by many to be the catalyst? Yes, the delivery of the new Information System. 

The delivery of such a transformational information system has been underway in several forms for a number of years, at scope gathering levels, at business development levels and now at development and implementation stages. A significant amount of effort went into the delivery of what one supplier called “the Carlsberg of requirements specifications”. Regardless of this plaudit the day after the contract for delivery was signed (with a different supplier) it became clear that a procurement ready set of business requirements now needed to become a design specification, a logical data model and a set of business story boards. 

How to go about this whilst avoiding “overfitting” has become the challenge, particularly against a backdrop of organisational change and flux. Guarding the requirements whilst ensuring that what is being built is still relevant is key. Many organisations report that the answer to this problem is to build within an Agile methodology, but we were not geared up to be able to do this, we need to keep doing the day job whilst the new system is built. So our answer has been to put in place the closest possible relationship between the supplier and the organisation, but, more importantly to ensure that “the business” (a differentia I detest) is as close as possible to all elements of the delivery. 

But where does that leave us, we turned down the possibility of delivering this system using Agile as we were worried about the resource intensity, but, our solution has quite probably been as resource intensive as going down the Agile route. However what we have done, so far, is mitigate any “overfitting” as the resource working with the development and delivery function owns all elements of the requirements specification and therefore can maintain their own confidence that what is being built and demonstrated at each stage is what was specified originally and continues to meet the business needs. 

The guard against “overfitting” though is not only at the business resource level. It also requires a robust and sensible change management process that is managed and believed in by both sides of the contract. We achieved this for our delivery a little later than we should have, and initially we had issues caused by this, but now with our supplier leading us by the nose we have this process in place and it works well enough to facilitate a sensible conversation at all project delivery levels about the difference between a contractual change requirement and the technical elaboration of a business requirement into a coded piece of delivery. 

So, “overfitting” avoided, well we hope so, only implementation and business use will actually tell. As with the concept of “overfitting” in statistics if we have not avoided it we will have a system that delivers a double whammy of issues; a system that looks good on paper only and performs worse in the real world, issues that are very difficult to recover from, but we think we have them avoided, have you? 

Advertisements

To be a fox, or to be a hedgehog?

I am reading Nate Silver’s The Signal and the Noise, and a thought struck me about his categorisation of those that can make good forecasts and those that would be best leaving it to luck. He proposes that there are two types of people who forecast; the Hedgehogs amongst us are specialized people, often having spent the bulk of their career on one or two great problems; he then puts it to the reader that the Foxes amongst us are able to incorporate ideas from many different disciplines and apply them regardless of the origin or political spectrum.

When creating an information security team what do we look for, experts in their field or adaptability. I hate the phrase “Jack of all trades, master of none” personally as it comes with such negative connotations and when creating the team I wonder if that would be the better option. 

However if we play the Galileo of number crunchers supposition further forward then it starts to really test what do we need. A Hedgehog believes in the Big Idea and in governing principles, surely ideal qualities for a leader in the Information Security environment (Maybe even the whole Information Systems environment). A Fox is a scrappy thinker, able to believe and act on a plethora of little ideas, able to take multiple approaches simultaneously to any problem. So in this descriptive case it makes me think that the Hedgehog is the old school and the Fox is the new school, dealing with Big Data, Open Data, Governance Frameworks and Commercial relationships all at the same time. 

But, Hedgehogs make better television guests! They make big bold statements and inspire confidence, they have an air for the dramatic, which lets face it is how we often get Information Security on the agenda. But how often can we make big bold statements and be sure that the prediction we are making will come true, particularly if as Hedgehogs it’s a big bold prediction about our one big idea that we have had all of our careers. Foxes differ to Hedgehogs in this situation as they have that pluralistic approach to problems, often giving off an air of not being sure at all. However in a crisis the Fox will make better predictions of outcomes and will not be caught out chasing a false hope just because that has always been the answer. 

So, what with the Fox and the Hedgehog and the old fable of the Tortoise and the Hare I think I may need to open a petting zoo to house my Information Security team, I am convinced its about the balance of animal types but I also think I want to have a Fox with just some elements of Hedgehog as the leader. 

NB – The Signal and the Noise (The Art and Science of Prediction) by Nate Silver is a wonderful summertime beach read, full of ideas and bold statements! 

Putting the “IS” in team!

I have a directorate of around 50 people that I am extremely proud of, they work hard to deliver some wonderful innovative solutions to quite complex problems against a financially “tight belt”. The structure is quite traditional for any informatics directorate;

  • Project Management function
  • Technology Office (Including Infrastructure, Development and Test)
  • Design Authority (Including Business Analysis)
  • Service Management
  • Business Intelligence Unit

Once every quarter we have an away “day”, its more like four hours than an actual day but it gives the whole directorate chance to get away and work together across disciplines on issues and problems, building relationships and developing solutions. It normally entails a bit of a social side too!

This quarters away day had two key parts, firstly a day in the life of the CIO, it would seem that the team wanted to know what it was like to be me, and a session in the style of the TV programme Dragons Den which would allow groups to work up a pitch on an innovations solution they thought we could bring to the organisation. My Senior Management Team (SMT) acted as judge and we promised there would be a prize.

So, for me the first part was difficult, a day in the life of me, I can’t just stand up and say, jeez, it’s busy! Nor did I want to get into the detail of everything that happens, some days are exciting some days go to plan and schedule (thanks in part to the very excellent PA I have). So I put down my list of things to do and my list of things I worry about, I make a lot of lists, and I used this to try to get across the complex ins and outs of running Information Systems for clinical research in the UK. It seemed to work, not too much “consultant bingo” was one comment and the other was no wonder we are all so busy if you have all that in your head!

Next was to kick start the Dragons Den, we had set aside a good chunk of time with a lunch break in the middle to allow the teams that the SMT created to create their ideas and pitches, we had provided resource to help develop thinking and of course, being an Informatics team there were copious amounts of technology on the desks to help.

At the beginning of this blog I said I was proud of the team, and here is one of the reasons why. They were put in teams they would not normally work in and within 5 minutes there was an intense buzz about the room, they were given tight deadlines and scenarios but jumped into it like men and women possessed, and at the end of the session we had eight original, benefit led ideas that, if we had the time and resource, I would implement into the heart of our business this year. The diversity of the suggestions was remarkable, ranging from portal based learning development tools to Information Security toolkits, quite mind blowing.

But the winning suggestion, and not because it had the trendiest theme, was a BIG query solution, a solution to open up unstructured documentation (to the tune of half a million documents going back to 2006) to enable learning and best practice to be got at from across the process of gaining permission to do clinical research or run a clinical trial. Not only was this at suggestion stage, the pitch was well thought through and was absolutely grounded in the benefit it could release to research in England.

Not just the “dragons” but the entire directorate believed in the solution and could immediately see the benefit, so now we are in the early stages of how to do this, the wining team leading the delivery of this new innovation.

So, the next time someone says to me that IT is complicated; it doesn’t understand the business or is for geeks and techies I am going to insist they come to one of our away days and see a modern IT team in action, a team that understands the business and puts benefits before technology. We have achieved this simply by working together.

Implementing Information Governance and Security – A Piece of Cake

The National Institute for Clinical Research (NIHR) Clinical Research Network (CRN) is the clinical research delivery arm of the Department of Health. Today the organisation is governed through around 70 hosting contracts across the NHS, in April 2014 this will reduce to 15 as the CRN implements a transition programme that will simplify the structures for clinical research in the NHS.

Having oversight of Information Governance will be easier and implementing frameworks that can be adopted for Information Security will become something that can be audited more thoroughly, but with change comes risk. New contracts, new ways of working, new staff and new monitoring arrangements. The organisation funds around 10,000 staff who work on clinical research and ensuring they have access to training and tools to protect the organisations and participants in clinical research is a huge “piece of cake”.

The organisation is a network of structures and with this in mind we have implemented information governance and security through the availability of:

Best Practice – Ranging from training through to templates for key elements

Steering Groups – Resource structures to provide support to IG leads

Frameworks – Audit frameworks to provide assurance

Enabling the organisation to learn its own lessons in a safe environment has been a goal of the last 12 months. Reducing the risk but allowing each element of the structure to evolve its own SOPs has been important to ensue that each part of the structure has ultimate buy in. Utilising tools like those demonstrated by the Analogies Project have been particularly valuable when attempting to explain to an academic researcher why Information Security and Governance are so important.

What is the key lesson as we have learnt though as we move at significant pace towards the new structure?

“If people believe in the outcome they will help implement security and governance.”

We have spent the time explaining the why it is important without turning our resource into extras for the Spooks TV series and now they understand Information Security good practice – it is becoming second nature. As an organisation we have moved from the Department of Health’s audit tool categorisation of work to do to satisfactory in 12 months, and this is down to two things, the buy in and the access to the expertise at the ISF.

All in all, Information Security is all about getting buy in to eating the piece of cake!

Everything is BIG, the data and the challenge…

The National Institute for Health Research is charged with improving the health and wealth of the nation through clinical research. That’s no small task, and cutting edge Information Systems are key to achieving it. Not only do we need systems that reinforce the NIHR’s position as the most integrated health research system in the world, we also need to embrace concepts such as Big Data and Open Data – ideas capable of making “UK plc” the destination of choice for running commercial clinical research studies. But how do we make sure we make the right choices, and what does this mean on a day-to-day level, and how on earth do we secure all that data?

Innovation in information systems has been pushed hard at the health economy in recent times, with only some success. The average nurse on the ward is bored of being promised something new that will make life easier, so this time we have to deliver. Six and a half thousand NHS staff are involved in research activity, so the challenges include how to help these people to collaborate, how to avoid duplicating work, and how to ensure that conducting research at the point of care is as easy as it possibly can be whilst being secure and of a high quality.

Our role is to make the NHS accessible to researchers, but what is the first thing we need to do with Information Systems to do this? When we ask that question, we are told: simplify them, make them faster, make them bigger, make them more secure, make them portable. So are we innovators or prioritisers?

In the last five years, the number of participants involved in research in England has trebled, and the systems we deploy need to handle growth of data at such an exponential rate. However,data in itself is not enough. We also need to transform data into information, and then create the means to gather insight from that information. This is now a demand of the researcher, but also the research participant. Tell me what my involvement in research has achieved, is now a legitimate question that anyone conducting research needs to answer, and therefore we need to provide systems to facilitate it.

Does this lead us towards a massive corporate strategic infrastructure, one big solution for lots of needs and problems? Emphatically, no. What we need is the ability to provide interoperable systems, link legacy systems to new shiny systems, and utilise open data standards. We have tried to use that word “open” in a different way by opening up our data to information managers across the NHS, allowing them to create open queries that can be shared across the organisation, and providing to the catalyst for service improvement. So far it’s working!

…. but the next challenge is how to secure open data, what do we need to do with it and how open is it really… All questions we will start to answer over next six months.

Thanks for reading, if you want to know more get in touch or follow me @r1chardatron.