What Problems Are We Facing in Solana? A Deep Dive with SolanaFM

10 min readJun 2, 2022

Introduction

We are almost halfway into 2022 and we have witnessed several battle-tests on Solana so far. After numerous network outages, the community has been doubtful of Solana’s network stability. Protocol Exploits and ‘rug pull’ incidents have also thrown into question the robustness of the blockchain’s security. In addition, Solana developers recently reported a discrepancy between the time on the blockchain and that of real-world time.

Amidst all the uncertainty, SolanaFM aims to zoom into the factors that gave rise to these issues, and whether they have a major impact on the Solana ecosystem. This report will elucidate three prevalent issues within the blockchain; Congestion, Security and Time Discrepancies.

Congestion Issues: Overload of Transactions by Arbitrage & Mint Botting

This year alone, Solana has faced 8 network outages, mainly attributed to automated spam transactions for both arbitrage trades and NFT mints. Let’s dive into each of them in more detail.

Arbitrage Botting

An arbitrage is the process of buying a digital asset on one exchange and selling it almost simultaneously on another exchange where the price is higher, with the difference being the profit. The transaction below visualizes the movement of funds during an arbitrage trade — A user swaps USDC for acUSD on Saber and makes the opposite swap on Mercurial Finance, taking home almost $50,000 in profit.

Sample Arbitrage Trade Transaction. Source: SolanaFM

Arbitrage opportunities like this are common when price movement is volatile. Since transactions in Solana are cheap, traders can make use of bots to submit thousands of duplicate transactions in order to snatch the most profitable trades.

This was what happened on 21st January — traders made use of bots to spam arbitrage trades when the market dropped almost 20%. These duplicate submissions also prevented other users from fulfilling their transactions.

The aftermath of this network outage could impede the growth potential of Solana’s DeFi ecosystem as both users and protocols are affected. Users were liquidated from their positions, and affected protocols like Solend had to reimburse users who were affected by the abnormal volatility.

NFT Mint Botting

When a user mints an NFT, they convert digital data into a unique digital asset that they can exchange on an NFT marketplace. The transaction below visualizes the movement of funds when minting an Okay Bears NFT — The user sends 1.5SOL and a small fee to the creators to execute the NFT Candy Machine Program and create a new account for their NFT. Currently, Okay Bears is sitting at a floor price of over 150SOL.

Sample NFT Mint Transaction. Source: SolanaFM

Depending on the popularity of the project, NFT minting can be very profitable. Since the number of NFTs are limited within each collection, traders can make use of bots to spam mint transactions on the network — eventually minting multiple NFTs and ‘flipping’ them in marketplaces like Magic Eden and Solanart for larger profits.

On 1st May, a record 4,000,000 transactions per second were submitted on the Metaplex Candy Machine program, resulting in a 7-hour network outage. The overwhelming volume of bot transactions meant that users had almost no chance competing against the bots to mint their NFTs.

Apart from the negative implications on Solana’s network stability, bot transactions also paint an inflated image of actual NFT demand in Solana. These botting incidents could repel users from investing in future NFT projects, ultimately hindering the growth of Solana’s NFT ecosystem as well.

Solutions to Solana’s Network Issues

In response to the congestion issues, Solana developers have rolled out a few changes in their v1.10 update on the Testnet network:

Moving from the current data transfer protocol, UDP (User Datagram Protocol), to QUIC (Quick UDP Internet Connection) which offers asynchronous communication and reduced latency.
Integrating stake-weighted transaction processing, which is a rudimentary implementation of QoS (Quality of Service) that would prevent unstaked or staked bots from consuming large amounts of bandwidth.
Integrating fee-based transaction priority, where users can optionally submit transactions with additional fees to put their transaction at the front of the queue.

In addition to these changes on the network, Metaplex has implemented a penalty on its Candy Machine Program for wallets that attempt to submit invalid transactions. For instance, if a bot sent out 100,000 transactions during an NFT mint and successfully minted 100 NFTs from completed transactions, the other 99,900 invalid transactions would incur a penalty.

These changes on will be able to speed up Solana’s transaction processing and sniff out bot transactions when it happens, thus preventing future congestions from occurring.

Security Issues: Rugs & Network Exploits

In almost all Layer 1 blockchains, scams and rug pulls are almost inevitable. As we build on the blockchain, more operational loopholes get exposed. On top of that, it is very difficult to dox (identify) users by their wallet addresses. These factors give hackers and scammers a favourable opportunity to exploit both the network and the community. Here are some notable incidents that have occurred across the NFT and DeFi markets.

NFT ‘Rug Pulls’

On 6th February, founders of Solana NFT project Balloonsville openly pulled the rug, scamming thousands of minters over 5000 SOL ($450,000) after being featured on the Magic Eden’s launchpad.

The anonymous founders revealed that the rug pull was carried out on the premise of berating Magic Eden for their lack of due diligence when selecting projects to be launched on their platform.

Shortly after, the founders of Balloonsville also announced that they part of another project, Doodled Dragons, which also openly pulled the rug on 9th January.

These projects are among many other projects including Big Daddy Ape Club and Undead Apes, which typically play the same cards before pulling the rug — fake partnerships with other NFT projects and artificial engagement on their social media platforms.

Throughout the ordeal, the identity of these founders were never revealed as they operated behind their official project accounts on Twitter. Since it was almost impossible to track the founders, this meant that they could carry out the scam more than once without getting caught.

If these scam NFT projects can launch so easily, how will we know which projects we can trust?

Network Exploits

On 23rd March, the stablecoin protocol Cashio was exploited for a staggering $52.8 million. Due to a flaw in Cashio’s code base, the hacker managed to mint 2 billion $CASH tokens with two billion of his own unknown tokens. The figure below shows the transaction where the unknown tokens were minted.

Transaction of hacker minting 2 billion unknown tokens. Source: SolanaFM

Due to the flaw in Cashio’s code base, the mint field on the arrow account was never validated. This allowed the hacker to forge a chain of fake accounts and mint the 2 billion $CASH tokens. The hacker then swapped the $CASH tokens to wipe out almost the entire treasury into USDT-USDC LP tokens worth $52.8 million. (For more details on the Cashio hack, read Bybit’s post-mortem here)

Another exploit occurred on Wormhole Network on 2nd February, when $320 million was wired out of the protocol. The hacker was able to bypass a signature verification step in Wormhole’s smart contracts, effectively being able to mint Wrapped Ethereum (a tradeable version of Ethereum on Solana) without depositing Ethereum themselves. Take a look at the transaction below to see the movement of the Wrapped Ethereum.

Transaction of hacker minting 120,000 Wrapped Ethereum. Source: SolanaFM

Solutions to Solana’s Security Issues

In order to combat these security issues, auditing services like Sec3 (formerly Soteria) have been designed to provide security analysis for Solana smart contracts. These services detect common security vulnerabilities, helping projects in Solana avoid costly security issues like the case of Cashio and Wormhole Network.

Magic Eden also made systematic changes to its marketplace. They developed a much stricter vetting process that requires the project founders to dox themselves (to the staff), and sign a legal contract in order to use their launchpad services. Read the full update here.

Being able to pin-point any dangerous flaws within the protocols and conduct strict vetting in NFT marketplaces can prevent future occurrences of exploits and anonymous projects pulling the rug.

Time Discrepancies: Clock Difference

Before we dive into the Clock Difference issue, a quick primer on slots. At essence, slots refer to the time taken for a leader (validator) to produce a block. As per the Proof-of-History (PoH) mechanism, slots are ordered roughly sequentially and non-overlapping, representing a decentralized clock that synchronizes with real-world time. Although the ideal slot time is about 400ms, clock drift allows blocks to take up to 720ms to be produced.

Last week, validators reported that cluster time had fallen behind due to a sharp increase in slot time. The average slot time was found to be over 750ms, much beyond the 720ms upper limit. According to Solana validator @laine_sa_, validators currently cache about 100 on-chain programs as needed (JIT). However, Solana is seeing over 1000 unique programs each day and validators have to constantly recompile these programs that are not cached.

The figure below shows a real-time comparison of the latest block produced (left) and the actual UTC time taken online (right). At this time, the reported Clock Difference was about 2 hours and 33 minutes.

Real-time comparison of the latest block produced and the real-world UTC time, showing about 153minutes difference.

Clock Difference is mainly a cosmetic issue, which does not affect Solana’s network operations or transaction processing. However, it could possibly contribute to reduced stake rewards. Solana’s stake rewards are paid by epoch, and at the ideal 400ms slot time, there would be 182 epochs a year. This means that an increase in slot time would result in fewer epochs being created, and thus fewer earnings for delegators and validators on the network.

Solutions to Solana’s Clock Difference

In order for the cluster to catch up to real-world time, slot times have to go back below 720ms. An easy change to make is to bump up the cache to compile more programs. However, compiled programs consume more RAM. Hence, this scaling has to be done cautiously to provide enough breathing room for the cluster to drift its clock back towards real time.

Alternatively, validators can make use of Switchboard’s ‘sysclock offset oracle’ which, in essence, tracks the clock difference and offsets the time on-chain to synchronize with real-world time.

Another Network Outage?

Yesterday, Solana faced its 8th network outage of the year. However, no programs were congested; instead, a bug was found on the network. According to Solana Founder Anatoly Yakovenko, a bug within the durable nonce instruction caused part of the network to consider the latest block invalid, hence no consensus could be formed and validators stalled. In response to the bug, the network has restarted with the durable nonce feature temporarily disabled while Solana developers devise the safest and most efficient solution for the bug. (You can keep track of the latest Solana releases on Github.)

This shows that there are still plenty of improvements to be made to the infrastructure. However, we also noticed that this outage saw the shortest time taken for validators to coordinate a restart so far — across 1800 existing validators, the network restart took 4 hours and 30 minutes. To put this into perspective, the network outage in September 2021 lasted about 17 hours across 800 existing validators. The time taken to achieve consensus is almost four times faster even after the validator community has doubled its size this year. While we cannot rule out the possibility of more bugs being identified and more network outages taking place, the active response of the validators and developers shows that the Solana ecosystem is on the right track.

The Bottom Line

Although the incidents discussed are unfortunate, the protocols affected are still absolutely necessary for the ecosystem. It is important to build on these infrastructural flaws to prevent similar cases from happening again. With services like Solana Pay reaching out to the general public and Solana’s daily active programs steadily increasing, the infrastructure of Solana has to be prepared to scale proportionately to handle incoming users.

In order to continuously innovate, projects in Solana have to be made more open-source. For instance, programs in Solana are available in the Solana Program Library on Github, allowing developers to step in and suggest improvements like increasing fees for writable accounts to tackle botting on the network (Read more on issue #21883 here). These contributions will lay the foundation for stronger infrastructure, preparing Solana for mass adoption in the near future.