By Max Colchester and Jessica Hodgson

LONDON--A computer glitch that left millions of Royal Bank of Scotland Group PLC (RBS) clients out of pocket underscores the challenge that banks face as they try to offer cutting-edge online services to attract customers while keeping cost as low as possible to appease investors.

As the RBS computer debacle crept into its eighth day Wednesday, the part state-owned bank faced continued scrutiny over why it botched what industry insiders called a routine software upgrade.

Over the past week, millions of RBS and NatWest customers were unable to access their money after a software failure meant that funds weren't credited to their accounts. On Wednesday, RBS said the issue had been solved but that 1% of all affected transactions were still affected. RBS said that some of the 2 million customers of its Northern Irish Ulster Bank unit were still experiencing "unacceptable" delays.

The meltdown at RBS highlights a technological conundrum for banks: rely on sturdy but dated banking infrastructure or splash out on necessary upgrades that could cause disruption for clients.

With the development of online banking, transaction volumes have increased dramatically, leaving bank's payment infrastructures--which often date back to the 1980s--creaking under the pressure, said Daniel Mayo, an analyst at consultancy Ovum. Today the technology used "is mostly aimed at trying to keep the lights on," Mr. Mayo said.

In the U.K., the problem is particularly pronounced as the banking sector has undergone a substantial consolidation in the past two decades. As banks absorbed their competitors, they were left to integrate a slew of back office IT systems. Many of these software programs were created in-house and poorly documented. As a result, when workers were fired or retired incoming staff struggled to get to grips with them.

Adding to this problem is the fact that banks have been conservative about upgrading their IT systems because they fear the negative publicity that a computer glitch could cause, said Chris Skinner, chairman of the Financial Services Club, a discussion forum for the banking industry.

RBS is a case in point. In the 1990s, NatWest was considered to have a state-of-the-art payment clearing system. However, when RBS bought NatWest in 2000 it decided to graft the lender's payments system onto its own more basic platform, Mr. Skinner said. The computer glitch that caused RBS' clearing platform to fail last week likely came as the bank tried to upgrade its system to cope with an increase in payment volumes, industry experts said.

The upgrade went wrong, leaving RBS to work back through millions of transactions in chronological order. A spokesman for RBS said that a software upgrade caused the problem but that the bank is still to determine what went wrong and why. The bank denies that 36,000 job cuts at the company affected the bank's technological know-how. Fully upgrading RBS' payments system could set the bank back by around GBP500 million, Mr. Mayo said.

Research firm Gartner Inc. said that U.K. banks budgeted EUR750 million to bolster their payment systems in 2012, up 7.7% on the year before. However, the improvements can take several years to complete and are fraught with potential risks.

"To do an infrastructure overhaul in banking, it's more like a heart and lungs transplant than in any other industry," said David Silverstone, delivery and solutions director, NMQA, a software testing consultancy.

For instance, when Spanish bank Santander UK PLC integrated customers of Alliance & Leicester in 2010, a number of clients complained that they couldn't access their online accounts.

In the mid-2000s, HSBC Holdings PLC (HBC) launched an ambitious program to create an international platform for its various global subsidiaries. The program called "One HSBC" stumbled when it came to knitting together aspects of the bank's international retail branches. Their different setups and product offerings were difficult to meld, and the software proved costly and unwieldy for the end-users. The program was ditched about a year ago. HSBC declined to comment.

Some banks bit the bullet and have rebuilt their IT from scratch. U.K. bank Nationwide embarked on a roughly GBP1 billion overhaul of its IT systems in 2008. The bank will fully complete the project by the end of 2014.

To offset potential problems, Nationwide will pay to run a parallel banking and savings ledger so that no customer information is lost in the event of a glitch during the changeover, said Tony Prestedge, the chief operating officer at Nationwide. "In our job, the most important thing is resilience," he said.

-Write to Max Colchester at max.colchester@wsj.com