Watches of Switzerland CEO Brian Duffy: “Never take any situation for granted”

Christopher Jackson meets an impressive business leader at the top of his game, and finds out how he did it

When I think back on all the people I’ve interviewed across many sectors, I sometimes think I see a single common denominator of what made them successful: time. All of them, from the hedge-funder who lamented about our interview (‘I’m going to have to set aside my time’) to the internationally famous businessman who introduced me to the phrase ‘hard stop’ understood its all-encompassing importance. All of them made sure they made every minute of every day count.

Watches then are more than just a luxury item – they symbolise our relationship with our most precious commodity. A watch is a thing of substance; it is luxurious but it is very far from trivial. And the best in the business is Watches of Switzerland, led by its redoubtable CEO Brian Duffy.

Duffy tells me about his upbringing, which didn’t necessarily seem to signpost him towards a luxury career. “I am from a very much working class background in Glasgow,” he tells me. “I lived in various housing schemes in Glasgow and attended a regular state school. Growing up, I couldn’t even have told you what luxury meant to be honest. Mappin & Webb have been in Glasgow forever and I do remember looking in their window and saying to myself: “How could anybody ever afford to go in there and shop?” It’s now part of our portfolio.”

I say that a person in his position might easily have gone the other way and harboured resentment at high end products. Duffy is frank: “I tend not to be envious: I feel today – and I think I felt it back then – that people are entitled to spend.  The desire to own a watch is a couple of hundred years old and jewellery several thousand years old.  As a species I think we have always enjoyed having these symbols of emotional investment and financial investment.  I never harboured negative feelings of envy or anything.  In fact I probably had more aspiration and desire than resentment.”

So did Duffy have mentors to help put him where he is today? I half-expect a story of a charming Glaswegian teacher who saw his potential early – but the opposite turns out to be true. “No – there was nobody. But I always had ambition and a personal confidence that I could be successful.”

Duffy then is a self-made man. How did his career start? “My route into what became my career was simple. Back then, you had a careers officer that you did 20 minutes with – and the careers officer asked me what I particularly liked. I was very much a generalist at school. I was good at languages and science, and I also loved history and the humanities. Since I was particularly good at maths and science, they said: ‘You should be an accountant’.”

As so often with successful people, there was a bit of luck involved. “It turned out that I was in the very last year where you could start accountancy in Scotland straight from school.  After my year intake, you had to be a graduate. I did a five-year apprenticeship that included one year full time at Glasgow University but the rest of it was part time education. That meant I became a chartered accountant very young because I started my apprenticeship at 17: at 21 I was a chartered accountant, qualifying with KPMG as it is now.”

After a short stint at Polaroid (‘they were a big employer near Loch Lomond’), Duffy was headhunted into the ladies’ underwear sector. It was to be a fast rise. Duffy became CFO of Playtex at 28 and then European CFO at the age of 30, which meant a relocation with his wife and three young kids to Paris. “My accent was very thick Glaswegian Scottish and so nobody could understand what I was saying.” Evidently, he made himself understood one way or another, as he was soon promoted to worldwide CFO of Playtex at 34, a move which precipitated a family move to Connecticut.

Back then Playtex was, says Duffy, a heavily indebted business. This turned out to be an opportunity: “I got the chance to invest in the business as a management buyout which turned out well. After three years they sold the business to the Sara Lee Corporation at a handsome capital return and it changed our life financially. Up until that point we were borrowed to the hilt with mortgages and credit cards: but from that day onwards I never borrowed another cent.”

Duffy moved across at the same time to general management, and found that by nature he was “more of a brand marketeer naturally than an accountant.” The family wanted to return to the UK, and so Duffy returned as CEO of Playtex. In 1991, he managed to negotiate the return of Wonderbra to the Sara Lee Corporation which led to a famous ad campaign with Eva Herzigova which I can remember growing up: I would have been 11 at the tme.

It is a tale of success upon success – and all carried on very young. By this point Duffy was financially secure but continued to work. I ask him about his motivation to keep going. “I enjoy work. I enjoy leadership. I enjoy accomplishment. I should have been retired ten years ago but I am still working as hard as ever. Besides, I feel a strong sense of responsibility to both shareholders and colleagues.”

All this is not to say it’s been plain sailing. Duffy had a fallout with the Sara Lee Corporation (‘a big strategic disagreement’) and decided to go to the Guildford Academy of Contemporary Music to play guitar during the career hiatus which ensued. “Whenever I talk to students, I always say: ‘If you find yourself between jobs as I did, do something productive with your time, and find something that makes you feel good about yourself every day’.  To me it was music. I told myself this was a wonderful opportunity of getting good at guitar that I would never have otherwise. I had played it all my life very badly so I really used the time productively and kept my spirits high. If you use between time wisely, it also means that you have good judgment: you won’t be inclined to jump on whatever comes along because you want to get busy again. If you are doing something you enjoy and you are filling your time, you are going to have to think about what comes along, which is what happened in my case.”

So did Duffy keep in touch with the outside world? “I did. Eventually what came along was Ralph Lauren and the opportunity to be the president of Ralph Lauren in Geneva. So I never finalised my degree in contemporary music but I got the first year of what was a two year course.”

Ralph Lauren marked the second stage in Duffy’s career. “I had ten years at Ralph Lauren, but I was travelling like crazy and wanted to be more settled.” It seems to have been a very happy time. “I loved those ten years working with Ralph, and getting to know fashion was a wonderful education.   I spent a lot of time with Ralph discussing product and being mentored by him. I lived in Switzerland for a year so there is an element there of understanding the Swiss and what makes them so accomplished at what they do.”

The time Duffy spent in Switzerland would help him understand his next opportunity with Watches of Switzerland, which came up through private equity in 2014. “We were a £300 million business when I started and today we’re almost £1.8 billion equally spread between the US and the UK. We listed on the stock exchange in 2019 and so we are now a public company.”

So to what does Duffy attribute his success? “Like everything you have to meet it with the same positive attitude when things don’t go your way,” he tells me. “All the way along, I had a healthy dose of personal insecurity and it was probably my biggest motivating factor. All the way through I wanted to make sure that I proved to myself first of all that I was going to be successful. I would never take any situation for granted. With the  role I am in, you have got to work harder and be more hands on and more committed than anybody if you are the boss. I think part of that comes from this disproportionate need to constantly prove your worth.”

I mention that he seems to be responsive to events, and skilled at waiting to see what comes along next. “I never had a five-year plan,” Duffy agrees. “Fortunately the opportunities did come. I was never in a rut at any time. Most of it was right place, right time. But looking back, in those companies I would always be pushing like crazy to the point where I cringe: I was always banging on the door either for promotion or for more money – probably for more money more than anything. I would be in there saying: ‘Look at that I am doing. Look at the value you are getting. I am not getting my fair share’.”

So would Duffy advise young people to emulate this approach?  Not exactly. “When you are negotiating salary or you do have a problem and you think you are not getting fairly treated, really think it through – really rationalise it. I always say: ‘You are probably among 90 per cent of the population that don’t think they are fairly treated. So I’d suggest you really test it in your mind and when you have got a strong rationale for it, whether it’s the market or the work you are doing or the accomplishments you have or whatever, then come in and present them as objectively as you can. One way or another, keep your boss on side.’. If you are pushing your boss and they say: ‘I hear what you are saying, and I want to help,’ then that’s a good situation to be in.  You are partnered on it together and you will get somewhere but if you say, ‘That’s not good enough. I want more,” and you tick them off, you are going to find it very difficult.”

At the other end of the scale, what advice does Duffy have about being a CEO? Duffy explains that it’s important to realise your blind spots. “You take the world I am in today. The buying of beautiful watches and jewellery is a really fun thing to be doing – and the selling is a really fun thing and the marketing of it and advertising. But as a CEO you are responsible for everything: you cannot completely delegate accountability. It’s not for everybody and you never escape.  If you do want to go home and switch off completely then probably being a CEO is not the ideal role.”

Does Duffy have any tips about hiring and the best way for business leaders to go about that? “Fundamentally, to be good as a CEO or a senior person in any organisation comes down to judgment: that’s where your success or failure is going to come from. Are you going to have the experience and instincts and a blend of the two to make those right calls?” Duffy says it’s also important to look at past mistakes. “Part of success is auditing what you have done in the past. I hired that guy and it never worked out. How did I miss that? How did he manage to convince me he was worth hiring? Why was I misled by his charming outward personality and I never got behind what was there?  You just have to learn from that and go back.”

Duffy adds that your retrospective analysis should also work the other way: “Equally look at your success stories and say: ‘Okay – that worked out well’. I still get things wrong. Some people really do interview very well and then turn out to be different. The great thing about the areas I am in –consumer goods and retail – is that they are great sectors for genuine meritocracy because it’s all about the numbers, right down to the question of why a given shop did better in the afternoon than in the morning. At Ralph Lauren, I discovered that the most valuable time to get the sales people together was immediately after the selling season – that is, just after they had been through the programme of selling. That’s because they were very raw but enriched with the numbers they had made or something they believed in that never came through.  The value you get from their judgment at that point was at its highest.”

Does his background still inform his attitudes in business? “What I hate in life and businesses is people who feel entitled. I often say to young people when I talk to them, that I think the biggest advantage in life can be a disadvantaged upbringing because you don’t feel entitled to anything, you feel that you have got to fight for it and you will get half as much for being twice as good – but that’s okay. In the world of retail, the meritocracy is naturally applied from how people perform.”

But at the centre of it all is Duffy’s passion for the product. “I love what we sell.  If I could summarise my talent, I do understand the psychology of the consumer and I can match the product to what I know will be the consumer preference at the end of the day. Watches are 90 per cent of what sell.   Its more or less a Swiss monopoly. People often say: ‘Why are people spending £10,000 on a watch when they could get one for a couple of hundred?’ This misses the point completely: I call it rational indulgence. Functionally, you could buy something that would do the job a lot less but you are buying a product that has got intrinsic value that will retain value and could well appreciate in value – and in addition to that, it’s liquid. If you ever decide you get tired of it or want to do something else with it then you could sell the product. “Inevitably people don’t buy watches frequently, but will buy them when something special happens in their life: a graduation, anniversary, or business success.  Once I’ve understood this, I then need to ask myself: ‘What are the trends with regards to case size or dial colour or materials or functionality?’  They might appear straightforward, small products but there is a lot of different things that can represent the watch functionally and aesthetically.”

It’s infectious listening to someone talking about what they love – and it’s infectious listening to Duffy discuss the history of watches. “I can’t imagine any other culture in the world could have done in the world of watches what the Swiss have done,” he enthuses. “And it all came down to John Calvin, who was the de facto ruler of Geneva when it was a city state. He would tell everybody how to lead their lives: he was very austere and he banned the wearing of jewellery which he said was ostentatious and frivolous. So that all of a sudden, goldsmiths and silversmiths had a product they couldn’t sell – and so they turned their hand to watch making and clock making.”

So what does the future hold for Watches of Switzerland? “We have had phenomenal growth but we have equally said to the stock market that we plan to double our business again in the next three to four years. But it’s really focused on high growth in the US: we feel the watch market in the US is underdeveloped. Not only that, but we have already proven it to be the case and have created a billion-dollar business in six years.”

So which parts of the US is he referring to? “We are in 18 states in America.   Our head office is Florida where our biggest concentration is. The next would be New York and the next would be Las Vegas. Bur we are also in Cincinnati, we are in Minneapolis, and we are in California in a small way. We are in Colorado, and Connecticut.   We are dotted around but there is a lot more that could be done to increase our presence.”

He is realistic but relaxed about the prospect of a Trump administration: “Obviously you have Donald Trump saying he will be imposing tariffs and so on – so that obviously would have an implication on everything that would come into the country. But we deal with an aspirational consumer. The big thing in America is that everybody says in the year of the election the consumer pulls back and as soon as the decision is known in November they start spending again whichever direction it is. People can handle anything except uncertainty.”

That would presumably mean that the last decade in British politics hasn’t been all that helpful either? “Brexit affects the market and clearly what happened in the latter years of the Conservative administration impacted the market.  Because of Brexit we pulled out of VAT free shopping and we are the only country in the Western world that doesn’t offer tourists the chance to shop VAT free. In our category that just seems like madness; it makes us uncompetitive.”

And all this uncertainty has culminated in Rachel Reeves’ budget. What’s his reaction to the early moves of the Starmer administration? ”Inevitably if people have less income, one way or another it’s going to impact how they spend. The most important thing is some degree of stability and certainty. We all do well when the country is doing well and you hope for peace and stability and a better balance in the country, which I do think is what we can expect from this government.”

It’s a remarkable career built on a phenomenal work ethic, a wry sense of humour, and an increasing commitment to social purpose (see our article on the King’s Trust overleaf). “I am managing a company today I am proud of. We have a broader sense of responsibility and our foundation is a big part of that.  We have dedicated resources to environmental matters as well, which we were doing beforehand and there is a natural pressure as a public company to do that.  We believe as a company you have got a bigger role in society than just looking after your shareholders: having that proper balance and sense of responsibility is very important to me and our organisation.”

I find myself pleased to have met Duffy, and to have understood a little more of what it takes to build a successful career in this sector.  His goals over the next year are highly ambitious but given his resolve, imagination and quiet authority, I certainly wouldn’t bet against him achieving them – and more.