Insights – City Short Courses

Category: Insights (page 1 of 3)

Megan O'Reilly

London Culture Shocks from an Irish Perspective

by Megan O’ Reilly

I’d always had a feeling that I’d love London. From the bustling tourists to the stuffy Tube, every element contrasted with the small Irish town I’d called home for 23 years. Despite feeling ready for my move across the Irish Sea, I didn’t anticipate the culture shocks I’ve come to know so well.

Adjusting to life in a new environment can be difficult for the best of us, and perhaps I was a bit naive when I gathered my belongings and headed to a city of ten million, coming from an island with a population of half the amount. I’d lived abroad two years before – in Bologna for my Erasmus year – and thought myself well versed in new experiences.

Stepping off the plane at Stansted was something I’d envisioned since I’d left school, and after settling into my new house in Twickenham and getting to grips with the trains (who knew tapping in and out could cause such grief!), I started to see the small differences between my Galway and London lives. This was in 2019.

London

Galway

I started working at Fortnum and Mason as a Retail Host. The considerable gap between social classes soon made itself apparent in the 300-year-old luxury establishment, something that isn’t as obvious in Ireland. I found it difficult to understand how such wealthy people could live and shop only a few roads away from a growing homeless population finding shelter in Green Park Tube station.

I noticed how much more diverse London was – a very positive aspect which I admire a lot. My colleagues and customers came from a range of backgrounds and cultures. They made working there not only refreshing but extremely interesting. While Ireland is slowly becoming more culturally diverse, there’s still a long way to go.

As Irish twangs go, the Galway accent is on the softer side, and when I first started working in the capital, I was asked what part of Canada I was from! I wouldn’t consider my accent particularly strong, but of course, I received affectionate comments from my London friends on how I pronounce certain words.

Two years on, I’ve adjusted some of my slang. I now say “trainers” rather than “runners”, and when asking hypothetical questions I’m conscious of using “Shall we?” not “Will we?”. You’d be surprised how quickly people pick up on the smallest of differences.

As much as I miss Ireland, I’m glad I left. London has so much to offer, and I’m ready for even more new friends and new excitement! I’ve started making a list of at least 20 new restaurants to try out, and have given Soho’s bustling bars great business since the end of lockdown. And yet there’s a reassurance in knowing that I’m never far from another Irish person.

I often think of a time when I was on holiday in Spain with an English friend, and we bumped into a lovely lady from Dublin. It was St. Patrick’s Day and any and every Irish person was celebrating. My friend couldn’t believe I’d stopped to talk for five minutes with a stranger. Apparently, she’d never do this if she met someone British abroad. This sense of community across the globe is in our Irish blood.

For anyone thinking about taking the leap to pastures new, I can’t recommend it enough. Diving into a new place in which to discover yourself and flourish is a fantastic experience, and something I believe everyone should strive for. As scary as it may seem, you might just find yourself in your new home. By being in a new environment you allow yourself to escape your comfort zone. Just expect a few surprises along the way!

Megan completed our Introduction to Copywriting masterclass with Maggie Richards. For more information on our writing short courses visit our home page here.

If At First You Don’t Succeed…

September 21, 2021 / Emily / 0 Comments

The Road Less Travelled: City short course alumnus Simon Culleton’s long journey to publication

By Simon Culleton

‘I know a literary agent,’ said my opponent as we passed at the net. I tried to act casual to disguise my eagerness so waited until we had played two more games and passed again. I feigned breathlessness.

‘Perhaps,’ I said still catching my breath, ‘Perhaps you might want to put in a word for me.’ He sucked the air through his teeth and looked as though I had just asked for one of his kidneys. He waited until we were stood in front of a crowd of people on the clubhouse veranda before counting off three fingers.

‘One, I’ll need a letter of introduction, he said. ‘Two, a brief outline of what the book is about and three…’ he now had the complete attention of a gathering crowd, ‘And Three, I’ll need the full manuscript with no spelling mistakes.’

‘What, no spelling mistakes?’ I didn’t say that of course, I just accepted his request with a subservient bow of my head. I’m a writer, and like all writers am desperate to get published.

I’d love to tell you that I let him win the tennis match, but he far outranked me and was always going to win. I had only agreed to play with him because I heard he had a friend who was a literary agent.

It had been three years since I’d first walked into the classroom at City University of London’s ‘Novel writing’ evening class. One of the first tasks that our tutor, Martin Ouvry, had set for the class was to document why we wanted to write our chosen novel. It was a telling exercise.

My answer was honest; I didn’t want to write this novel, I wrote. It was too personal and raw. More accurately, I continued, ‘the last thing I wanted to do was remember. Yet inevitably, almost fatally, whenever I attempted to write a different storyline, all my characters were either divorced or battling in some way for their children. So eventually I submitted. It was always going to be ‘Shadows of Fathers’ first.

I remained with City and progressed to their year-long Novel Studio course. I enjoyed the twice-weekly structure and the twelve-thousand word, deep critique was a particular landmark in my novel progress.

The Novel studio course paid particular attention to obtaining an agent worthy to champion our book. Emphasis was put on presentation, catchy letters to attract an agent:

“Dear Madam, I respectfully submit… Dear Sir would you please consider… or Dear Michael I read in your bio that you enjoy stories that surprise you… Hey Sarah, like you I play tennis (badly) …

I sent over fifty, all of which got nowhere, most didn’t bother replying. I even tried some of the foreign literary agents. A reply email from Hamburg went something like this:

Thank you for your story, Simon. Everyone in the office really enjoyed it although the literary agency no longer owns these premises, we are boat engineers.

I stayed with City University and enrolled in a further three workshops with Katy Darby as well as travelling to Greece for the Athens international School Of Creative Writing. One particular highlight was attending a flash fiction class taught by the excellent writer Heidi James.

I quite literally immersed myself in the writing world. Although I had yet to find representation; a nagging doubt that was always with me. One of the hardest things I found about writing a novel is that you have to finish it before knowing whether it will be a success.

During the first lockdown, I became despondent until a chance text conversation from an old friend I had not seen since my school days. (When we were young teenagers she had let me hold her hand at the bus stop). ‘I know someone who is a publisher’ she texted. A sudden vision of the man standing on the tennis club veranda came into mind. But this was Bernadette, I thought. I had missed a bus for her when I was fourteen.

As it turned out, my tennis friend didn’t

Author Simon Culleton

know an agent, after all, he only knew the father of the agent and had subsequently fallen out with him, (possibly over a spelling mistake).

So once again I sent off my synopsis and the first fifty pages. After a few weeks, I received a request to send the rest of my novel. I was on top of a wobbly tower scaffold laying heavy blocks when I received an online zoom invitation. Rose Drew of Stairwell books, an American woman from Florida whose exuberant hand gestures took up the whole of the computer screen, was enthusiastic. She had read my book and could relate to all my characters and recite any passage from my novel. I had found my champion.

It has been a long and arduous road with weekends and evenings spent writing in libraries and coffee shops, London university corridors and crowded Greek restaurants. At work I was forever scrawling notes for my novel on pieces of timber and newly plastered walls; conversations were cut short while I retained a thought later to be added.

It takes dedication and sheer bloody-mindedness to complete a novel and in my case a lot of help and guidance too. City was a wonderful place that helped harness my book idea to the finished debut novel that is Shadows of Fathers.

About the author: Simon Culleton was born and bred in Essex England, where he lives with his two children. His love for writing began when he wrote a short story at age 17, while sat in a derelict car, which went on to be broadcast on BBC Radio 4. He loves to travel and has worked his way around the world, undertaking jobs from snow clearing in Sweden, to construction work in California. Simon has a passion for chronicling everyday people which extends even to himself: he has maintained a personal daily diary for over 40 years.

About the book: When Richard realizes his German wife is not returning to England with their children, the subsequent journey he must take encompasses new geographical and emotional realms. With the help of comic but effective German lawyer Otto Lehmann, Richard’s fight for his family is both heart-wrenching and humorous, in a story that crosses countries and cultures. Shadows of Fathers offers an alternative view of separation: a dedicated father fighting for the right to parent in a new and relevant take on contemporary fatherhood: not only in the mid-1990s setting but also in today’s society.

Simon’s debut novel, Shadows of Fathers

Shadows of Fathers is available for pre-order on Amazon, Google books and many more. Published by Stairwell Books in October 2021, the first chapter can be viewed on the ‘Coming Soon’ page at Stairwell Books.

Ten questions about Ten Steps to Us

August 26, 2021 / Emily / 0 Comments

Ahead of the publication of her debut YA novel, Ten Steps to Us, author and Novel Studio alumna, Attiya Khan, kindly found time to answer some questions from Novel Studio Course Director, Emily Pedder.

How did the idea for Ten Steps to Us first come about?

The opening scene started as a simple writing exercise in a creative writing workshop that I was part of. It got really good feedback in the class and the story just blossomed in my mind. The characters of Aisha and Darren and their forbidden love developed over time into a full novel.

Did you always know you wanted to write a Young Adult novel?
Attiya Khan, author of Ten Steps to Us

I have three teenage kids and I remember my daughter saying there were so few books that she could relate to because there were so few BAME characters. This really spurred me on to write such characters. I also think the angst and the pain you feel as a teenager, when you are learning who you are as a person, are so interesting to write about as there is so much conflict.

Which writers or books have inspired you?

I love all kinds of books: thrillers, crime, romance and literary fiction. At the moment I am a little bit obsessed with Elif Shafak and am working through all her books. I recently finished The Forty Rules of Love which is about Sufiism, and I thought it was a masterpiece.

Your novel deals with complicated issues of religion, race and class in an accessible and entertaining way. Was that important to you when thinking about writing this novel?

It was very important to me. The book is not an autobiography but there are elements of myself in Aisha. As a Muslim girl growing up in Kent I often felt very isolated and that I didn’t quite fit in. I am very interested in the angst that people feel when they are caught between two cultures. I wanted to play with the idea of being stuck in the middle and the confusion and pain that brings, when you don’t quite know which way to turn. I wanted to convey that it’s okay to be confused about who you are and to feel torn. You don’t have to be perfect to be a Muslim. Religion is a very personal thing, and it really is between you and God. Everyone finds their own way.

What’s your writing process? Do you plan first, or do you write to find out what you want to say and how you want to say it?

A bit of both to be honest. With this book, I wrote the beginning and then the end. I got stuck in the middle so did a chapter plan and worked out what was going to happen and then wrote it.

You’re a graduate of City’s Novel Studio. Can you tell us a little about your experience on the course and how it fed into your novel?

The plan that I just talked about was suggested by the Novel studio in one of the tutorials and it really helped. I probably wouldn’t have completed the first draft of the novel if it wasn’t for the Novel Studio. We did a showcase and following this an agent contacted me saying she wanted to read the rest of the novel. This motivated me to complete the novel.

What was your path to publication? And how has the experience been so far?

The agent who contacted me following the showcase didn’t actually sign me, but her interest led me to believe in myself a little more. I completed the first draft of my manuscript after she showed interest and shortly after this was selected for David Higham’s Open Day for Underrepresented Writers and then longlisted for Undiscovered Voices 2020. I really started to believe in my novel after this. I got picked up by Hashtag Blak, a publishing house for underrepresented writers at the start of the pandemic. It has been a great experience so far. We have been through several rounds of edits and the book is out for publication 9^th September. Exciting times!

Amazingly, you’re not just a novelist but you’re also a GP and a mother of three. How do you find time to write?

It has been very challenging, especially with the pandemic when work became very, very busy. However, for me, writing is an escape from the stress of day to day life. I also got a lot of support from the critique groups that I was part of. It is hard but I love it and I guess that’s what kept me motivated.

What advice would you give your younger writing self?

Believe in yourself! Listen to constructive feedback and take it on board but also develop a thick skin. Other people’s opinion is very subjective, what one person hates another person may love. Tell the story that you want to tell.

What are you working on next?

I am in the middle of writing a GP thriller. (Again, I must stress this is not an autobiography!) It’s about a doctor with mental health problems who is working with another doctor who she suspects is a murderer. Is she losing her mind or are her suspicions correct – and if they are correct how will she get anyone to believe her?

Thank you so much, Attiya! And huge congratulations on your brilliant debut. We wish you all the success in the world with it.

Ten Steps To Us will be published by Hashtag Blak on 9th September 2021. Pre-order the book here.

‘To Write is Human, to Edit Divine’: Why it’s Essential to Edit Your Copy

August 16, 2021 / Emily / 0 Comments

By Hannah Boursnell

Stephen King’s tribute to his ‘divine’ editor – from his book On Writing: A Memoir of the Craft – is a creed all writers should live by. Yet when I recently turned my red pen on my own writing on a City short course, I was reminded that even professional editors need to keep their words in check.

Words are seductive. You select one ‘perfect’ adjective, add it to your carefully constructed sentence and put down your pen, satisfied. But then another flutters its eyelashes at you from over by the thesaurus.

You know you shouldn’t, but the temptation is too strong. So you add another. And another… Before you know it, you’re drowning in description and your word count is out of control.

Over the course of my 15-year career as a book editor, I’ve seen this problem rear its loquacious head time and time again. And on City’s excellent Introduction to Copywriting course, I came to the uncomfortable conclusion that even I’m not immune to the come-hither allure of an extra adjective.

My name is Hannah and I’m addicted to words.

The novelist Ernest Hemingway – renowned for his spare, efficient prose – offered this typically pithy advice to writers: ‘Use short sentences.’ He understood that every word must earn its place. Dense, over-elaborate text risks becoming tedious and impenetrable, but when writing is precise and simple, each word shines with a clarity of meaning.

If you suspect your writing might be overcrowded, the answer – always – is to edit. In her memoir Stet: An Editor’s Life, Diana Athill described editing as ‘removing layers of crumpled brown paper from an awkwardly shaped parcel and revealing the attractive present it contained’.

When editing, I do so with Athill on my shoulder. My pen is used to gently clear away anything that might obscure the author’s intended meaning.

But when I’m writing? Sometimes the joy of being creative on my own terms is so intoxicating that I’m prone to forget the editorial truths I hold sacred. One such truth is that a writer should always thoroughly edit their own work. This requires bravery, but you’ll become a better writer if you persevere.

Five steps to self-editing success

Take a break before you begin. Even an hour will provide a fresh perspective.
Edit on paper – you’ll notice things you overlooked on screen.
Change the font to trick your eyes into thinking they’re reading something new.
Read the text aloud – especially dialogue. It encourages weaknesses to reveal themselves.
Cut as rigorously as you dare, but save previous versions. Just in case.

‘Kill your darlings’… with kindness

I’ve found that the practice of self-editing is both sharpening my writing and making me a more compassionate editor.

When a draft has been loved and laboured over, every cut can sting – whether the edits are made by a professional or with your own red pen. Deleting my own precious words, I’m constantly reminded of the courage and vulnerability that’s required any time a writer puts pen to paper. It is a privilege to be entrusted with another writer’s work.

I’m not sure if I’ll overcome my addiction to words, but they say that admitting you have a problem is the first step. In the meantime, I’ll continue to pack my first drafts full of delightful, dazzling, delicious words – and seek divine inspiration as I edit them.

Hannah Boursnell took City’s Introduction to Copywriting course which is taught by the brilliant Maggie Richards. For more information on all of our short writing courses, visit our website.

Narrative or Therapeutic Non-fiction: does it really matter?

July 8, 2021 / Emily / 0 Comments

By Raviakash Deu

Doctors, nurses, scientists have all played their roles this past year, but for narrative non-fiction writers, what does it mean to serve on the front-line? I knew, without really knowing, the answer to that for some time. When done well, any writing grounded in the facts as much as in the imagination has a way of inspiring, energising and in some cases healing the minds of its readers. On deciding to pick up the pen, one begins to take control of those transformations not only in others but crucially within oneself.

I experienced this phenomenon in company and in spirit on City’s Narrative Non-Fiction Short Course, which might just as easily be termed ‘Therapeutic Non-Fiction’ – and not just because the group boasted a psychologist. In the wake of artistic absence in the world, City’s virtual offering brings together traditional storytellers, reporters and scholars from across the globe who, while seeking guidance on an outer narrative, inevitably end up fulfilling part of their inner one too.

Creative writing is a soul-bearing business. Setting weekly classes in the digital sphere might present students with additional complexities against the larger editorial goal of stripping them away. Yet this hasn’t stopped City, who’ve plenty of reason to trust in tutor-extraordinaires like Peter Forbes. As one of those under Peter’s stewardship over the last three months, I’ve been glad to convert a fairly demanding hobby into a more thoughtful practice, and beyond that, develop a confidence that had been desperately missing prior to week one. Starting out as a nervy penman amongst some sophisticated scribes, by week eight, I presented my changed state in the following journal entry:

‘A rare environment. One which appears to value personal growth, indeed community, over competition. In reading aloud our compositions, it’s a unique opportunity to bring alive the material for a bright-eyed audience, of which I too am an avid member. The talents of the group are unlike anything you’d expect. I sink into Roli’’s delicate depiction of makeshift graves on the banks of the Ganges, Monica’s rich reflections on ‘room-travel’ and Roz’s masterful musings on imitation, a lyrical style translated effortlessly into her diction. There’s Imran’s artful sketches on humans in the age of machines, and Robert’s endlessly entertaining travelogues.

The academics, meanwhile, seem to keep us honest. Both Katherine and Claire are as faithful to their subject areas as they are to the business of elegantly unfolding them for us mere mortals. Oh, and amidst all this, I’ve perhaps discovered my own capacity for spinning a good yarn.’

‘Good’ writing is, of course, subjective, and arguably, my hopes going into the programme were of unearthing something ‘real’ rather than ‘good’, a readiness – as spoken by Hesse – to ‘gaze into the fire, into the clouds and as soon as the inner voices begin to speak… surrender to them’. It’s thanks to City’s new expression of ‘bookbinding’ or a sharp sense of literary unity, I’ve been able to take meaningful strides toward that free and fearless outlook, and all its potentialities.

Raviakash Deu is a freelance writer from Birmingham. He holds an undergraduate degree in English Literature from the University of Nottingham and a Master’s in Shakespeare Studies from King’s College London. His regular features appear at ‘The Lipstick Politico’ where he is interested in bringing light to daring South Asian narratives across culture and the arts.

Narrative Non-Fiction runs on Tuesday and Thursday evenings from October 2021.

City Writes Summer 2021 Writing Competition

April 26, 2021 / Emily / 0 Comments

City Writes Summer 2021 Competition Opens

City Writes, the showcase for Short Courses creative writing talent, is back on Zoom this Summer with alumna, Alex Morrall as our professional. Alex’s debut, Helen and the Grandbeeswas published by Legend in 2020 to great acclaim.

Described as ‘Uplifting’ by the Daily Mail and ‘Breath taking’ by Awais Khan, Helen and the Grandbees is a mother and daughter reunion exploring identity, race and mental illness. We’re delighted Alex will be sharing the novel with us on Wednesday 7th July.

Alex is a Brummy artist and writer living in southeast London who took a Freelance Writing short course at City. She has had poetry published in several journals and writes food reviews for her local newspaper. You can find out more about her work on her website.

Alex Morall, author of Helen and the Grandbees

For your chance to join Alex on the online stage, all you need to do is send us 1,000 words of your best creative writing (fiction or non-fiction, YA but sadly no poetry or children’s fiction) by Friday 11th June 2021. Full submission details are here.

If you want to register for the event on Wednesday 7th July, you can do so here and if you are keen to catch up on the online events you’ve missed, check out the blog for links to videos and articles.

Don’t forget to send your best 1,000 words by midnight Friday 11th June 2021. Competition winners will be announced in week 9. We can’t wait to read your submissions.

Facing the fear of career change: from data analyst to copywriter

April 15, 2021 / Emily / 0 Comments

After years of feeling directionless in my NHS job, I’ve finally found a calling that’s reignited my passion. Here’s how I’m pivoting in mid-life thanks to a City short course.

by Christopher Hunt

For five years I’ve considered changing careers. As a Data Analyst for the NHS, the prospect of changing not only to a new career but from an employee to freelance feels daunting. I have a mortgage and two children under 12 after all. And yet, while it’s easy to make excuses, I’ve realised the only way to confront my fear is to act.

Writing has always been part of my life. I’ve self-published a supernatural novel, written guitar-related blogs and even scripts for a short-lived YouTube comedy series. I also have a fascination with psychology. Searching the internet for jobs related to these interests I discovered a career called copywriting. I could be paid to write!

‘Introduction To Copywriting’ by City University runs over a single weekend, fitting conveniently around my job. The course is taught by author and copywriter Maggie Richards. One of the first things she said is from novelist Ernest Hemingway: “The only writing is rewriting”.

I love this quote because it can be interpreted in different ways. While on the surface it’s telling us to rewrite our work until concise, it also encourages action – to start writing and overcome the ‘fear of the blank page.’ We can refine our work later.

This encouragement to move toward the unknown resonates with my aspirations: the initial steps toward a new career are similar to the first tentative words a writer must put on the page. Many of my doubts and insecurities are really just fears of the unknown.

As author Seth Godin says in ‘The Practice’: “The career of every successful creative is… a pattern of small bridges, each just scary enough to dissuade most people.” Much like the act of writing allows a writer to clarify their thoughts, it’s by taking action that we can find our next step and the step after that, slowly lifting the fog that obscures the path ahead.

City’s online workshop offered many opportunities to take action with practical copywriting exercises, working individually and in small groups. One of my favourite was writing home page copy for an imaginary app.

My team came up with ‘Fitness Friends’, where users meet new people sharing similar fitness goals:

Headline: Meet, Motivate, Get Fit.

Introduction: Walk, Run, Gym. Meet your goals with new local friends.

Call to Action: Find Fitness Friends Now.

Maggie pointed out that the headline and introduction used the same three-part staccato punctuation. I realised the importance of varying the rhythm of the words, blending punch with flow. Creativity should not cloud clarity.

I’m now taking steps to start my copywriting career alongside my NHS job, hoping to eventually become a full-time copywriter. I’ve signed up to a freelancing website and contacted small business owners in my network, offering my writing assistance.

Writers spend so much time living in their heads that it can feel uncomfortable taking physical action. But the pages of my life so far don’t have to dictate where my story goes next. Realising that no first draft is perfect, I now know I can shape my path until I reach the outcome I desired all along.

Christopher Hunt took City’s ‘Introduction To Copywriting’ course. You can find him on LinkedIn.

City offers short writing courses in everything from short story writing to writing for the web and digital media. To find out more about all our writing courses, view our full range here.

Cut Short: the debut book from short course alumnus Ciaran Thapar

April 1, 2021 / Emily / 0 Comments

This summer, Penguin will publish Ciaran Thapar’s debut book about youth violence, Cut Short. Novel Studio course director, Emily Pedder, caught up with him to find out more about his path to publication and the book David Lammy has described as ‘honest, authentic and raw’.

Emily Pedder: Your book, Cut Short, is an urgent look at the UK’s serious youth violence epidemic. What first drew you to this subject?

Ciaran Thapar: As an education and youth worker in schools and youth clubs in London between 2015-2018, I came to see how youth violence was playing out on the ground as an overwhelming force in the lives of young people I was working with. I therefore noticed that the myopic, if-it-bleeds-it-leads way the British media were reporting on fatal stabbings across London and beyond was not only detached: it was actively harming communities who were suffering most by telling stories that distorted, rather than informed, wider society’s understandings about social breakdown. So, moved by my interest in writing and journalism, I started to write about issues which I saw as orbiting the violence: austerity, school exclusions, drill music.

Ciaran Thapar’s debut, Cut Short

What’s more, 17-year-old Michael Jonas, the older brother of my first mentee, Jhemar Jonas – who I first started working with in January 2015 when I was a postgraduate student at LSE – was stabbed to death in November 2017. Quite suddenly I was required to become a consistent source of support for Jhemar and his family. It made me double down on trying to understand and solve why violence happens among young people. The book was primarily borne out of this motivation.

And then, in the hot summer of 2018, with London simultaneously celebrating the weather on the one hand while becoming weighed down by unprecedented death and tragedy on the other – divided by lines of race, class and postcode – I started writing my proposal. Cut Short’s creation is therefore moved by my practical journey as a youth worker learning the ropes, seeking to make impact and forging empathy with young people, as well as my intellectual attempts to understand these last few years in British society, map out how we are all connected and responsible for civic unity, and provide a hopeful blueprint for steps forward.

EP: You’re a youth worker and a journalist. How easy was it to make the transition to writing a book?

CT: The researching, interviewing, planning and structuring of stories I wanted to include in the book between the summers of 2018 and 2020 – the start of the proposal to the end of the first draft – was never easy, but it was manageable. I went freelance in May 2018 after working for six months in London prisons, and for a while I was struggling to make ends meet financially. Dealing with the stress of that time was very difficult. But pressure makes diamonds: I wrote so much journalism to stay afloat as a result that I became more confident in my arguments and observations about themes I was seeing firsthand in my youth work: social exclusion, inequality, austerity and music culture. I have long been interested in writing long form journalism – I rarely read anything else – so in the end, on realising the book idea’s potential, it felt more natural to switch from writing lots of shorter pieces to focus and go long for the book.

The bit of the transition I found truly hard, however, is the impact it had on me psychologically. It is, I think, impossible to fully immerse oneself in a book-length project while sustaining other areas of life with any normality. I am very proud of the jump I made from youth work and journalism to the book, and what I’ve created in the end. But it has been difficult juggling everything. I’ve skated very close to burnout. Fortunately, my writing practice is now more patient and protected than it ever was before. I compartmentalise it on certain days and protect it to focused sessions, rather than letting it affect my youth work or personal life.

So, in sum: it wasn’t difficult to make the transition to writing a book, and it has enriched my life beyond belief. I’ve learned a huge amount and built a platform that will, I hope, allow me to advocate for change going forward. But, as with anything sustainable and meaningful, the transition came with obstacles and burdens that I’m only really starting to make sense of now. I feel a responsibility to say that here.

EP: What are the most important lessons you have learned in writing your book?

CT: First, you can’t rush or force quality, and when things don’t go to plan, it’s okay, because they’re not meant to. I became so stressed at first when interviews fell through or huge distracting events happened in my youth work in the process of writing the book. But I soon realised that these developments are a

Ciaran Thapar (photo by Tristan Bejawn)

prerequisite for anyone trying to craft a nonfiction story. Real life is not predictable, and it moves in imperfect directions, so writing a book that grapples with that truth will never go exactly to plan. I’ve learned to see this as an exciting and rewarding truism, and be grateful for it, rather than worry too much about stuff I can’t control.

A second lesson I’ve learned is that, compared to writing a piece of journalism, a book requires long-term immersion. It necessarily becomes a big, the biggest, part of your life. Giving it space, switching on and off from it, finding modes of self-care and people who you can be vulnerable with to support you, is the best way to keep in check. I feel like now that I’ve written a book, my writing practice is totally transformed for the better, and I’ll have that for the rest of my life.

EP: You studied Narrative Non-fiction at City with Peter Forbes. Did you do other courses before or after? And what was it about Peter’s course that you found most useful?

CT: I’ve not done any other writing courses, per se, but I think it’s important to credit the MSc Political Theory I did at LSE between 2014-2015 as an essential step for my career. It gave me the ability to consume and form strong moral arguments quickly. Ultimately, Cut Short amounts to a grand moral argument about the vitality of the state, civic participation and public responsibility. I make a case for compassion and empathy in spaces like schools and criminal justice where, as far as I can tell, these values are being systematically rooted out, often in the name of profit. Academic study gave me the critical thinking and language skills to forge this as a binding ideology. Peter’s course two years later then made me appreciate nonfiction writing as a creative craft in which every word counts. So then it became about making arguments and political advocacy fun, listenable, readable and colourful. The course also created the weekly pattern in my life of sharing my words with a group and getting feedback; it built my confidence. Without that pattern I think I would have struggled to put myself out there in the way I eventually did.

EP: What’s the most helpful piece of advice anyone has given you as a writer?

CT: Writing is like dancing: it’s best when nobody’s looking. I’ve heard these words spoken by the former Guardian journalist Gary Younge – who features in Cut Short! – a few times, and it really rings true now that I think about it more and more. Having a target audience and being self-conscious is to some extent important to guarantee quality and think about an aim for a piece of writing. But too much focus on who is reading your words, or how that makes you feel, can distract from the task of speaking truth.

EP: What advice would you give to someone starting out on their writing journey?

CT: Being a good writer is only partially anything to do with the actual writing: it’s also about being an adaptable, humble human being; someone who is willing to listen, learn and observe. Before you put pen to paper, create patterns in your life which give you freedom to think and be inspired. If you’ve got nothing to write about, then it doesn’t matter how good your language skills are: the words won’t flow. So focus on acting as well as writing. And then write every day in a journal. Once you’re ready to get your writing into the public domain, reach out to as many editors and writers as possible, offer coffees, ask for phone calls. Make yourself memorable. Pitch regularly, and if a pitch isn’t accepted, ask for feedback, embrace the learning, and take the opportunity to improve. Never see failed pitches as failures. (Some of my most successful articles were the result of failed pitches; Cut Short was the result of many!) And finally, stick to writing about things that you know intimately because everyone has a story worth telling and a perspective worth sharing, it’s just a matter of putting in your 10,000 hours trying to figure out how to articulate it. The only way to fail is by giving up, so don’t give up.

EP: Can you describe your route to publication?

CT: I was emailed by my legendary agent, Matt Turner, in the summer of 2018 after he’d read some of my pieces. We met at a pub in Brixton and he asked if I had a book idea. I didn’t, but the conversation, stretched over several meetings, quickly turned to my experiences as a youth worker and my aspirations to write something long form that could make a real impact on how youth violence is understood in British society. I was and still am surrounded by characters who are heroic and deeply inspiring, and I felt like their stories needed to be told in a book.

From that moment onwards, I spent 10 months struggling to write the proposal. Really struggling. But despite the delays, Matt stuck by me, editing whatever I sent him, sometimes binning it altogether and telling me to start again, and building my profile in his industry network. He didn’t need to remain loyal like this, or wait for me to get my act together, but he did, and for that I will be forever grateful. His supporting role was perfectly executed. So by the time we submitted the proposal in June 2019 – it took tens of draft attempts, many sleepless nights and many, many instances in which I thought I might just give up – we had a willing audience of editors ready and waiting to read the idea for Cut Short. In the end, I was pleased to gain 9 offers from publishers, and I chose Penguin Viking because I could see how passionate their team were about converting my ideas into a reality.

Then in August 2019 I got to work, slowly at first, and I was roughly halfway through writing the first draft in March 2020 when lockdown hit. I wrote the second half in lockdown, between March-July 2020, which was bizarre, but I think it worked well as I could pour all my time and energy into it. The book gave me a sense of purpose that I may have otherwise lacked during such a disruptive period for the world.

Since then, the book has gone through 10+ editing and proofreading stages, I’ve had to have many, many conversations with my editor, contributors, advisers and friends about the ethics and safeguarding of the story. I’ve worked with the main characters – particularly Jhemar and another young legend called Demetri Addison, who I used to mentor at his sixth form college – to make sure they are totally happy with how it all reads and represents them. And now I’m trying to enjoy the calm before the storm of publication in June. It’s very surreal getting feedback from early readers. I’m ultimately nervous and excited in equal measure.

EP: How will you celebrate the launch of your book if we are still in lockdown when it’s published?!

CT: The book comes out on 24th June, three days after lockdown is supposed to end. My 30th birthday is one week later. It’s going to be a special summer.

EP: What do you hope readers will take away from your book?

CT: Two things, which I state early in the book: an understanding of some of the problems which lead to youth violence, and a practical blueprint of some solutions.

Regarding the problems, I’ve chosen to focus on presenting and evidencing an argument which says that the British state is failing particular groups of young people, and this is why youth violence occurs. Austerity is to blame, but so is technological change and systemic discrimination in our public institutions, going back generations.

Regarding the solutions, I’ve tried to show through the stories I tell and analysis I present how we are all connected, if subtly, to serious youth violence, and therefore how we might think and act differently in future to collectively solve it. I want readers to connect with the story, understand these arguments, feel something and then act on that feeling; adults to care, young people to feel platformed and inspired. Cut Short is a call to action. It is not just a book, it is my way of making change and turning the armchair thinker into a frontline doer.

EP: What are you working on now?

CT: I run my own charity, RoadWorks, which I launch at the end of Cut Short. We explore music culture and social theory to support young people at risk of exclusion and violence. I also now write consistently for British GQ – I have a monthly column about youth and music culture online called ‘All City’ – and I’m working on one special long form piece for the print magazine this summer which I’m excited about. Otherwise, I’m doing more work behind the scenes on making sure the book’s publicity and promotion is where I want it to be. Watch this space!

Congratulations, Ciaran! We can’t wait for the book to come out.

Cut Short will be published by Penguin in June 2021.

Narrative Non-Fiction runs on Tuesday or Thursday nights for ten weeks.

Interview with Novel Studio alumna, Kiare Ladner

March 22, 2021 / Emily / 0 Comments

Kiare Ladner’s debut novel, Nightshift

Novel Studio alumna and tutor, Kiare Ladner, published her brilliant debut novel, Nightshift, in February 2021. Novel Studio Course Director, Emily Pedder, caught up with her to find out more about the book and her path to publication.

EP: ‘Your debut novel is set in a pre-pandemic London, in the nineties. Reading it now feels like entering a different country. How do you imagine London will recover in the years to come?’

KL: ‘London has so much kinetic urban energy. At its best, it’s a place where a person can have the freedom to be whoever they want to be (or are), and find others who are like them. What I hope change will bring is a city with more realistic rents for its workers. With affordable space for creative endeavours. With the arts right there, accessible, at the heart of it. A city revitalised by new ways of thinking in culture, economics and politics. An urban landscape that holds the thrill of the avant-garde alongside home gardens created to give nature refuge. A place that builds on the sense of community some have felt more keenly recently. And that always welcomes the immigrants we rely on. Even now, there’s a lot to appreciate about being here. The parks, the free art galleries, the brilliant hospitals, the possibilities for anonymity, the joys of simply wandering. . . When asked if I feel British or South African, my gut response is that I feel most like a Londoner.’

EP: ‘Meggie is a fascinating character, full of contradictions. She could so easily have been a passive character, with Sabine taking all the decisions, but it feels as if you’re showing us it’s Meggie who chooses what happens to her, and Meggie who has to deal with the consequences. Was this a deliberate choice from the beginning or did you need to consciously make her decisions more active?’

KL: ‘From the start, I was curious about the idea of wanting to escape the self, wanting to be other, and how far you can push it. During the writing process, I felt that Meggie was driven by this desire rather than acted upon. As a writer, I inhabited her in the way that an actor inhabits a character, and from there her decisions came intuitively. However there is one scene in the book in which she is less passive than I’d initially written her, thanks to an inspired suggestion from a beta reader. The changes were subtle but kept my narrative more in line with my vision for it. Beta readers are invaluable!’

Novel Studio alumna and tutor, Kiare Ladner

EP: ‘Sabine is one of those characters I feel everyone will recognise. That sophisticated, aloof person we all secretly aspire to be. How important was it to you to interrogate the personas people create and what lies beneath?’

KL: ‘This disparity is perhaps what first drew me to writing. Fiction allows us to investigate and express a less commonly portrayed sense of what lies beneath exteriors and dominant narratives. So I’ll probably be interrogating it forever…’

EP: ‘Where does a story usually start for you? With a character? A line of dialogue? A ‘what if’ plot question? A feeling?’

KL: ‘For me, it tends to start with a conundrum. Something that causes an itch in my brain, some question or situation I keep fiddling with. So the beginning is fairly abstract. Then if I give it time and space, scribbling and thinking, it tends to attach itself to a voice, and from there the story builds.’

EP: ‘I love how your novel taps into that complicated question of identity, particularly for those who live far from their native country. As a South African whose made London your home, is that an experience you relate to?

KL: ‘Definitely. I have gained a lot from being a stranger in a country, and the freedom to find my own tribe. But there are also aspects to leaving your country of origin that are painful, complex and irresolvable. Much to keep grappling with, in part through writing, I guess.’

EP: ‘You’ve studied creative writing at many levels, from short courses at City right up to PhD at Aberystwyth. What’s been the most important thing you’ve gained from that study?’

KL: ‘I’ve had some excellent tuition over the years. But I’ve also learned so much through other student writers. Not only from their brilliant and inspiring work – which has shown me the range and versatility of fictional prose – but also from their work ethic: their perseverance, resilience and determination.’

EP: ‘Do you think creative writing can be taught?’

KL: ‘It certainly involves craft, and learning. And a course environment makes space for a particular quality of attention to the work. I like how George Saunders puts it when he says that even for those, “who don’t get something out there, the process is still a noble one – the process of trying to say something, of working through craft issues and the worldview issues and the ego issues – all of this is character-building and, God forbid, everything we do should have concrete career results. I’ve seen time and time again the way that the process of trying to say something dignifies and improves a person.”’

EP: ‘How are you finding teaching on the Novel Studio, a programme you took yourself?’

KL: ‘Years ago this course gave me an inroads to the nuts and bolts of writing a novel. Its structure was invaluable in maintaining momentum and providing a sense of progression. And some of the other writers’ novels had me in awe! Now, what I find most exciting is to see the growth of the students’ writing over the course of a year. How hard some of them work, and how much they can do and learn and change. Also, the ways they engage with each other’s texts, their generosity in terms of time, attention and encouragement, is very heartening.’

EP: ‘What are you reading right now?’

KL: ‘I always have lots on the go in different genres (poetry, short stories, biography, comfort-for-the-middle-of-the-night etc). I’ve just excitedly added Mary Ruefle’s lectures Madness, Rack and Honey to my pile. And the novel I’m reading is This Mournable Body by the wonderful Tsitsi Dangarembga.

EP: ‘What are you working on now?’

KL: ‘A new novel called Skylight. I dare say no more!’

Kiare Ladner

Kiare’s short stories have been published in anthologies, journals, commissioned for radio and shortlisted in competitions, including the BBC National Short Story Award 2018. She won funding from David Higham towards an MA (Prose Writing) at the University of East Anglia, and then received further funding for a PhD (Creative Writing) at Aberystwyth University. She was given Curtis Brown’s HW Fisher Scholarship in 2018. Her debut novel, Nightshift, was published by Picador last month and is available to buy now.

For information on the Novel Studio and how to apply, visit City’s website.

For those who want to hear Kiare read from her novel, she will be the guest at our next City Writes on 1 April.

Register for free attendance here.

What really caused the Excel error in NHS Test and Trace COVID-19 system? An in-depth technical analysis.

October 13, 2020 / Dionysis Dimakopoulos / 0 Comments

Introduction

This is an in-depth analysis of the reasons that led to the COVID-19 positive results Excel error of the NHS Test and Trace system. The analysis is done using knowledge that a student can gain after studying a series of computing short courses at City, specifically Applied MS Excel, the series of VBA in Excel courses and the Database Design with SQL Server short course.

We have collated information published by the government and reported by news media to recreate, as faithfully as possible, the process that failed importing all COVID-19 positive test results.

We are also recommending steps that every company should follow when importing data from external partners, and the learning path prospective computing short courses students should take to gain enough knowledge to solve similar integration problems effectively.

Background

On Monday 5^th October 2020 UK newspapers were reporting of a technical error in NHS’s test-and-trace system. The error meant that more than 15,000 positive cases of COVID-19 infections between 25^th September and 2^nd October were not included in daily statistics and thousands of people who had come in contact with infected individuals were not alerted.

In this post we are going to focus on the technological aspects of the error. We will try to figure out what might have gone wrong, by putting together information published by the government and newspapers and will give recommendations on what you can do to avoid facing similar errors when importing third party data or integrating your systems with external partners.

Information gathering

We will base our assumptions on a note describing the methodology used for COVID-19 testing data, published by the UK government [gov.uk-note]. It appears that testing is categorised into four pillars. According to the Mirror [Mirror], the error happened while handling ‘Pillar 2’ data. According to [gov.uk-note], pillar 2 is testing for the wider population collected by commercial partners. The dataset for pillar 2 testing comprises of:

nose and throat swabs, which are counted together as one sample
tests counted as they are dispatched
‘in-person’ tests processed through laboratories, excluding the ones counted at dispatch
positive cases.

According to the note, there have been a couple of revisions to pillar 2 metrics and methodologies.

On the positive test results, which was the dataset where the error occurred, methodology was updated on 2^nd July to remove duplicates across pillars 1 and 2, to ensure that a person who tests positive is only counted once. Specifically for England, the lab surveillance system for pillar 1 and 2 results removes duplicate records by running a complex algorithm that identifies individuals and only uses their first positive result for the metric. The algorithm uses the following properties to uniquely identify an individual:

NHS Number
Surname and Forename
Hospital Number
Date of Birth
Postcode

News media presented a series of explanations of what is believed that had gone wrong.

According to Daily Mirror and Daily Mail, “Excel spreadsheet reached its maximum size” [Mirror] [Mail]
Daily Mirror also reports that “Outdated Excel spreadsheet format that was not capable of displaying all the lines of data” was the issue. [Mirror2]
Daily Telegraph [Telegraph] goes into more details: “The problem emerged in a PHE (Public Health England) legacy system. Public Health England was reportedly using an automatic process to pull the testing data it received from commercial firms carrying out virus swabs into Excel templates. But the old Excel file format being used – XLS – could only handle 65,000 data rows. The files have now been split into smaller multiple files to prevent the issue happening again”.
The Guardian [Guardian] on the other hand reports that the process is not completely automated and a lot of work is still done manually. It appears that CSV files are sent from labs to PHE, which are then loaded into Excel.
Finally, BBC reports that each test result created several rows of data. In the same article, there is also a comparison between the XLS and XLSX file formats of Excel, claiming that the new format would be able to handle 16 times more cases than the older XLS one. [BBC]

In depth analysis of what caused the COVID-19 Excel error

Public Health England has not yet published exact details of what went wrong. What we will do is to try and simulate what might have happened, by putting together pieces of information from the governmental website and news media reports.

To do so, we will create a dummy CSV file that contains the properties(fields) [wikipedia-csv] used as unique identifiers for each person tested, together with some dummy fields that represent test results. We will then go through the most plausible scenarios and discuss what could have gone wrong, to produce the error experienced by the NHS Test-and-trace team.

A CSV file is a text file that represents tabular data. This means that it contains a specific number of columns and one or more rows. According to the basic rules for CSV files [wikipedia-csv] and the 2005 technical standard RFC4180 which formalises the CSV file format, “All records should have the same number of fields, in the same order”.

This is an example of what data would definitely exist in the CSV file (first represented as a table and then in CSV format – Disclaimer: NHS numbers are random):

NHS Number	Surname	Forename	Hospital number	Date of Birth	Postcode
485 777 3456	Smith	John	HN3829904	12/03/2001	HD7 5UZ
943 476 5919	Smith	Jane		21/12/1958	HD7 5UZ

This is a CSV representation of the above tabular data:

NHS Number,Surname,Forename,Hospital number,Date of Birth,Postcode
485 777 3456,Smith,John,HN3829904,12/03/2001,HD7 5UZ
943 476 5919,Smith,Jane,,21/12/1958,HD7 5UZ

Further columns could be added to represent test results, but each row (record) should have values for each column (or at least simply a comma if a value is missing).

In order to test importing CSV files that are very large for Excel to handle, we created a dummy CSV file with 1,050,001 rows that has the following fields: NHS Number, Surname, Forename, Hospital number, Date of Birth, Postcode, Test number, Test result. The number of rows is larger than the limit of 1,048,576 rows that newer versions of Excel have [Excel-limitations].

The file contains random data that do not conform to data types of individual attributes. Specifically, the NHS Numbers generated are 10 random digits, where the 10th digit is not the control digit, postcodes simply follow the rule of having two letters-one or two numbers-space-one number-two letters format to look like postcodes but are not verified to be valid postcodes. You can download the dummy file from our Covid-19 Excel error analysis GitLab repository, where you will also find the Excel VBA code used to generate the test data.

Importing a CSV file that Excel cannot handle

Let’s try to import the generated CSV file into Excel. We do not know the version of Excel PHE is using, so we are going to go with the latest Excel 2019. News reports do mention that XLSX format could be used, so we assume PHE is using an Excel version after Excel 2007, but we are expecting similar error messages will appear in all Excel versions.

Opening CSV file directly in Excel

Here we see the error message we get if we try to open the generated CSV file directly in Excel. The way we opened it was by double clicking on the CSV file in the File Explorer, as the CSV extension is associated with MS Excel automatically during typical installation. An alternative way of opening the CSV file from within Excel would be to use the Open dialog, navigate to the directory that the CSV file is stored in and open the file from there.

The error explains clearly that when the user clicks OK, Excel will truncate the file and only show the part that fits the rows and columns available in one worksheet.

Importing CSV using Power Query (also called Get and Transform or Get Data)

If the user tries to use this new Excel functionality to import the CSV file she will be faced with the following error:

Again here we see a very clear error message, which explains that when the user clicks OK the data will be truncated and Excel will only display as much data as it can fit in a worksheet. Clicking Cancel will not import any data at all.

We see that both ways of opening a file in Excel, without using VBA code, show an error message notifying the user that data will be truncated. Clicking OK and continuing with only the data that fit in a worksheet is obviously human error.

Importing CSV using VBA in Excel

News reports mention that there is a (semi)automatic way of importing data in CSV format. Such automation can be done in many different ways. One automation could be that the user opens the CSV file normally and then, using a central dashboard, instructs Excel which worksheet represents the CSV file that was just opened and should be imported. A variation of this kind of automation could be that the user points to a Table in Excel as the input that represents the imported CSV file (a Table is created when Power Query is used to import a CSV file). Both of these scenarios expect the user to open the file with one of the ways we describe above.

Another way of importing a CSV file would be using Visual Basic for Applications (VBA) code in Excel. Again here there are many valid ways that VBA code can be written to import text files. In order to test this scenario, we created a VBA subroutine that reads a CSV file one row at a time. Each row that is read is split into attribute values and entered in the next available row of a worksheet. No error handling was implemented in the code.

Below you can see the type of error the user would get if the CSV file was imported via VBA code. This is the error message shown by the VBA interpreter:

This error message is definitely a lot more cryptic than the two errors seen above. The choice of buttons is also quite difficult to work with, by an untrained user. I am not sure whether the user would click on “Help” (only to get further unhelpful information – as shown below), or simply click “End” to stop the execution of the VBA automation. I am fairly certain though that either way the thought that first came to the user’s mind would be “HELP! I don’t know what to do.”.

In every way we see this, an error message would have appeared on screen, which means a user clicked OK without understanding the implications, possibly due to no relevant training. There is one possibility that the user importing the CSV file might have not been shown an error message. In this scenario, a VBA developer chooses to suppress all error messages shown from the VBA interpreter (like the one above). This is usually done either in an effort to avoid scaring the end user, believing that no error messages will be thrown by the VBA code written and if any is thrown it won’t affect the end result. In this case, human error is still the cause of the truncated dataset. However it is not the end user importing the CSV file that caused the error, but the VBA developer.

Remarks on the process

Storage structure of test results in CSV file

BBC [BBC] reports that each test result generates more than one row of data. We have two interpretations of what this could actually mean, based on the fact that data is delivered in CSV format.

Each test generates time based results, i.e. one value in 30 minutes, another value in 90 minutes etc. and the decision whether the test is positive or negative comes after a simple calculation between these values.
The process was misunderstood by the reporter. What really was meant is that in the same dataset there might be two tests (with two individual test IDs) for the same patient. This might happen if for example the first test became contaminated or a second test was done the same day for whatever reason.

As mentioned in Wikipedia “CSV formats are best used to represent sets or sequences of records in which each record has an identical list of fields. This corresponds to a single relation in a relational database, or to data in a typical spreadsheet”. The relational model used in relational databases and spreadsheets is most often represented as a table, where a header defines the attribute(field) names and each row has attribute values for each attribute name. In the relational model each row represents a unique record. This is the reason we are sceptical about the premise that a test result generates more than one row of data. Each row needs to be unique in some way, by a combination of attribute values. The use of a relational format to represent data that are not following the relational model does not make sense. This is how our assumption was made that each result must be unique either by including a timestamp or some other unique identifier or attribute, if two or more rows of the dataset are for the same test. On the other hand, we believe it is catastrophic if two rows cannot be uniquely identified as an individual entity, but still give two values for the same attribute.

Use of CSV for transportation of results

CSV is a very widely used format. It is not known when it was first created, but it already existed in 1972 [IBM-Fortran]. Even though it has been used for at least five decades, CSV support is varying across software. Its flexibility means that it is very easy to create CSV files that do not conform to all expected characteristics of CSV files. It is also very easy to break. A badly generated CSV file with the wrong value for one of its attributes, for example a comma to denote thousands in a number, i.e. 1,532.25, would not be imported correctly by any software, unless a different separator was used instead of a comma, a practice that is quite common. Usually the structure of CSV files is documented within a project, so that both the exporting and importing applications can correctly support the files generated.

Taking into consideration the limitations and old age of CSV format, as well as the potential duplication of data between multiple rows in the CSV file, we believe a different file format should be used (e.g. XML or JSON).

Use of Excel

There has been a lot of criticism on the use of Excel for COVID-19 test results, given that PHE already has a robust database, used for years, to collate test results for various diseases [Sky-news]. From this Sky News article we see that Pillar 2 data are probably the only data not directly sent to the database. It appears that Excel is used to open and upload the CSV data to the database.

Is the use of Excel valid in the case of getting COVID-19 test results from Pillar 2 privately-run labs and converting them and sending them to the main PHE database? We need to think of all the requirements and limitations that existed at the time of conception of this use of Excel, before we decide:

First of all, in March 2020, with the need to increase COVID-19 tests rapidly, privately-run labs were set up. We believe that each lab is using its own software to record test results. It is expected that most if not all of this software was able to export to CSV format quite easily, maybe with minimal set up.
Second, uploading data to any database needs to pass some validation, so that the database does not become corrupt. Such checks are best performed on the side of the database, instead of the side of the user – where user is each lab.
Third, new software needed to be created in almost no time to be able to handle the data sent by the labs. It would also need to be used by users that would require almost no new training. This means that an extension for a software that users already know how to use is the best option.

Excel is probably the software all PHE users knew how to use, in varying degrees, depending on their position. For time zero, a VBA extension in Excel seems like the first logical step. Excel VBA is commonly used as a rapid application development tool to test an idea.

VBA is a quite flexible language that, by leveraging the power of Excel, can help create very powerful extensions in very short time. We believe a very first version of a VBA extension that could handle CSV files sent by private labs could be created in a few hours, to handle the first data coming in, needing processing and uploading to the database.

Once a primitive way of importing data was set up, two parallel processes should have begun:

One should revise, expand and vigorously test functionality of the VBA extension, with a focus to eliminate human error from the process as much as possible.
The other should be to create an implementation that bypasses Excel all together and allows privately-run labs to use it to upload test results directly. A great way to do so would be through a restricted secure web service.

We believe that if the importing VBA process was correctly designed and tested, even an old version of Excel from 20 years ago could handle any CSV file size. The limitation of 65,536 rows that Excel has for each worksheet is not something that should stop an experienced VBA developer in creating a robust VBA add-in that can import CSV files of any size.

If the contents of the CSV file are converted by an Excel template to be uploaded to the PHE database, then the VBA procedure should read in memory one row of data at a time and upload it, instead of importing the whole file in a worksheet. This approach has two limitations. The amount of RAM available on the PC to hold one row of data in memory and the amount of hard disk space available to allow storing the CSV file. We believe that both of these are sufficient on the PC where the error occurred, given that it successfully loaded sixty five thousand rows into Excel.
If the user needed to view the raw data of the CSV file in Excel then, depending on the screen size, only about a hundred rows of data would need to be displayed at any one time. This can be achieved using a sliding window technique. Again, this is something that Excel could handle in pre-2007 versions, as it is far lower than the 65,536 rows available.

Our conclusion is that Excel was correctly used as a solution that satisfied all requirements at the time. A correctly designed and implemented Excel VBA add-in is also able to handle any number of rows from a CSV file.

What should you do to avoid this happening to your company?

Let’s explore best practices when importing data and integrating processes with an external company. If your company is collaborating with an external partner and prepares to import their data, then you need to have a bulletproof process to handle the incoming data. It is important to create an automated process and remove user involvement as much as possible to minimise or even eliminate human error. It is very important to test your automation vigorously, especially at edge cases and around known limitations.

If you are starting a new partnership and you want to test a satisfactory integration solution before implementing a full system that will cost a lot, Excel is a great choice. Most IT users already have some exposure to Excel. With minimal training you can train your end users to use VBA add-ins. Excel has grown and matured to become a tool that can handle any amount of data, limited only by system resources, provided that data is loaded judiciously.

You need a specialist that understands data, Excel, VBA and databases in depth.

What computing short courses will provide required knowledge?

A computing short courses student that has taken Excel, VBA and Database short courses will be able to design and implement a system that can import any amount of data from a CSV file into Excel and store it in a large database. Our recommended learning path would be:

Applied MS Excel for Business, where important functionality and correct use of Excel is taught, as well as limitations of Excel.
Visual Basic for Applications in Excel (Part 1, Part 2 and Part 3) three part course series, where students learn how to leverage the power of Excel and automate it using VBA, as well as import data from or integrate with external systems.
Database Design with SQL Server, where students learn how relational databases work; how to design a relational database and how to store data.

Conclusion

A robust automated system could have been created using Excel and VBA to handle importing of COVID-19 test results from CSV files of any size. Both Excel and VBA are able to handle this, if the automation is correctly designed, implemented and tested. A computing short courses student that has studied City’s Applied MS Excel for Business course, VBA in Excel series of short courses and optionally the Database Design course would have enough knowledge to design and implement such a system.

Furthermore, if end users of the NHS Test and Trace system were trained on the way the CSV importing automation works for COVID-19 test results from privately-run labs, they would be able to alert immediately that one of the CSV files could not be handled by the automation, saving precious time in the tracing of contacts of infected individuals.

We conclude that it was definitely human error that caused the COVID-19 positive cases to be missed, either at the user level while importing the data, or at a developer level where limitations of Excel were not taken into account. A well informed and trained Excel VBA specialist would be able to design and implement a CSV import and conversion system correctly.

References

[BBC] https://www.bbc.co.uk/news/technology-54423988, retrieved 10/Oct/2020.

[Excel-limitations] https://support.microsoft.com/en-us/office/excel-specifications-and-limits-1672b34d-7043-467e-8e27-269d656771c3, retrieved 10/Oct/2020.

[gov.uk-note] https://www.gov.uk/government/publications/coronavirus-covid-19-testing-data-methodology/covid-19-testing-data-methodology-note, retrieved 10/Oct/2020.

[Guardian] https://www.theguardian.com/politics/2020/oct/05/how-excel-may-have-caused-loss-of-16000-covid-tests-in-england, retrieved 10/Oct/2020.

[IBM-Fortran] http://bitsavers.trailing-edge.com/pdf/ibm/370/fortran/GC28-6884-0_IBM_FORTRAN_Program_Products_for_OS_and_CMS_General_Information_Jul72.pdf, retrieved 10/Oct/2020.

[Mail] https://www.dailymail.co.uk/news/article-8805697/Furious-blame-game-16-000-Covid-cases-missed-Excel-glitch.html, retrieved 10/Oct/2020.

[Mirror] https://www.mirror.co.uk/news/politics/16000-coronavirus-tests-went-missing-22794820, retrieved 10/Oct/2020.

[Mirror2] https://www.mirror.co.uk/news/politics/spreadsheet-blunder-meant-48000-potentially-22797866, retrieved 10/Oct/2020.

[Sky-News] https://news.sky.com/story/coronavirus-data-can-save-lives-data-can-cost-lives-and-this-latest-testing-blunder-will-likely-prove-it-12090904, retrieved 10/Oct/2020.

[Telegraph] https://www.telegraph.co.uk/technology/2020/10/05/excel-error-led-16000-missing-coronavirus-cases/, retrieved 10/Oct/2020.

[wikipedia-csv] https://en.wikipedia.org/wiki/Comma-separated_values, retrieved 10/Oct/2020.

About the author

Dionysis Dimakopoulos is the subject coordinator for the computing short courses at City, University of London. He has been teaching Visual Basic for Applications in Excel since 2003. He is an experienced software engineer, IT integrations consultant and published researcher. He has decades of experience creating systems that combine the power of web services with the familiar interface of Excel for engineering or financial applications, or offer interoperability with Office and other applications. His latest work is on the Learning Designer, an open online learning design tool for teachers in all sectors of education and subject areas, used around the globe, where he is the lead developer.