{"id":1520,"date":"2022-03-30T13:57:21","date_gmt":"2022-03-30T12:57:21","guid":{"rendered":"https:\/\/dovu.earth\/?p=1520"},"modified":"2022-03-30T13:57:23","modified_gmt":"2022-03-30T12:57:23","slug":"scaling-infrastructure-how-we-did-it","status":"publish","type":"post","link":"https:\/\/dovu.earth\/en\/scaling-infrastructure-how-we-did-it\/","title":{"rendered":"Scaling Infrastructure: How we did it"},"content":{"rendered":"\n<p><em>DOVU CTO Matt Smithies writes about our recent public performance test and shares insights for others looking to run similar experiments.<\/em><\/p>\n\n\n\n<p>In Q1 2022 we ran a public performance test on the Hedera Hashgraph testnet. We wanted to prove that we are able to scale in line with a vision.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What does vision mean to us?<\/h2>\n\n\n\n<p>By vision, we mean our ambitions. Not just surrounding our platform and staking work, but also DOVU\u2019s ability to generate carbon credits. In the future, we want to start utilising more third-party data and potentially Internet of Things (IoT) devices.<\/p>\n\n\n\n<p>At a glance these are the stats:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Over 30 million transactions in 24 hours<\/li><li>Averaging 1.25 million transactions an hour, the maximum was around 1.35 million transactions an hour<\/li><li>This averaged out at 247 transactions per second<\/li><\/ul>\n\n\n\n<p>It goes without saying that this kind of scale isn&#8217;t possible on the majority of current web3 technologies. We observed the content of various other chains, like Cardano, which ran <a href=\"https:\/\/finbold.com\/cardano-passes-the-20-million-transaction-mark-without-a-single-outage-in-4-years\/\" target=\"_blank\" rel=\"noreferrer noopener\">20 million transactions in 4 years<\/a>. This inspired us to run this test.<\/p>\n\n\n\n<p>This explainer will help you understand what it takes to reach these standards.&nbsp;<\/p>\n\n\n\n<p>As a note, we acknowledge that there are more entities out there that have pushed more throughput through the network. However, we feel that every single one of our transactions generated value rather than providing limited long-term value, by simply using the consensus service to send arbitrary messages to the network.&nbsp;<\/p>\n\n\n\n<p>In this test, each transaction was real data which had value connected to it. Throughout, off-chain storage manipulated this value every step of the way.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>The tl;dr<\/strong><\/h2>\n\n\n\n<p>Well, we achieved our goals. In fact, we surpassed them by approximately 20% across the board. Below are some images demonstrating the TPS we hit, resulting in over 30 million transactions in 24 hours.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh4.googleusercontent.com\/plWZMst1uOvVKCbtw4deENr1p6IoS-Au5564Jp4G2pFlDFHUeVnLZ_fW00Jx0I-PwzGz-Mpehive0cpiMASYLSoaMgsmQSgK99lfph3RjVwwxj68YUxE9JseBr-Obx-evgz-pkCH\" alt=\"\"\/><figcaption>Cumulative total transactions<\/figcaption><\/figure>\n\n\n\n<p>In terms of hourly transactions, we managed to max out just under the 1.35 million mark.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh3.googleusercontent.com\/c9ubcRs-_Qe8eSrNvKnD6GUgPZEXAku_NcQz3XI-f1jOc1BEziAQsWmlQRQ3dy-qAda3NEVnFjo-zYlzoc0MUjRmLC2gzgXgv0QIDAk0KSsFQgnSwacDxEvFJDNLrt-5bHg6wj7b\" alt=\"\"\/><figcaption>Transaction volume per hour<\/figcaption><\/figure>\n\n\n\n<p>We averaged around 247 TPS, although we did have occasional spikes of 300 and higher.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh4.googleusercontent.com\/bvpxfHKiSFCjC6aAKS_wJJ4fQPnGa6aTqggR2r5lE7UcdK4cJXdamqubvFsxBsCdhEzadtrU4-zmcIM4SKcn7CwG8DX75Rcj8KCC9QcjXkHwKj9F9WrAdjmY8Tel9HGMHtC-JQpD\" alt=\"\"\/><figcaption>Indication of average TPS<\/figcaption><\/figure>\n\n\n\n<p>However, it wasn\u2019t an entirely smooth process. We learned a lot. And we urge you to read on so that you too can learn more about our experiments for the network.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>DOVU\u2019s TPS history on Hedera<\/strong><\/h2>\n\n\n\n<p>Back in 2020 we first utilised the <a href=\"https:\/\/trust.enterprises\/\" target=\"_blank\" rel=\"noreferrer noopener\">Trust Enterprises API<\/a> (a tool I developed) to create a note using our proof of carbon mechanism. This was the first foray into building on Hedera and employing the consensus service to record every single transaction that happened within DOVU.&nbsp;<\/p>\n\n\n\n<p>While this was an achievement and it was easy to create, our initial aim was to record every transaction through our legacy system, almost acting like a side chain.<\/p>\n\n\n\n<p>Throughout this period we are able to push approximately 40-50 TPS on the testnet.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Understanding our current landscape<\/strong><\/h2>\n\n\n\n<p>As the marketplace and staking platform evolved, we needed to increase our capability and bandwidth to drive future transaction volume.<\/p>\n\n\n\n<p>The demands of ESG initiatives at a global scale are getting tougher. In terms of both audibility and demand from enterprises for verifiable carbon, the ability for us to scale quickly is a feature that we need to be capable of doing with lower downtime.<\/p>\n\n\n\n<p>Furthermore, as one of the pioneering staking platforms on native Hedera, which is completely non-custodial and lower risk, we need to expect an influx of users of at least 20,000 or more in the medium term.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>The plan of action&nbsp;<\/strong><\/h2>\n\n\n\n<p>In preparation, we set a couple of success metrics that we wanted to capture throughout the 24-hour testing period:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Averaging 1 million transactions an hour<\/li><li>Reaching 20-25,000,000 transactions in 24 hours<\/li><li>Having bursts to hit 250-300 TPS<\/li><\/ul>\n\n\n\n<p>But sending transactions to the Hedera network wasn&#8217;t the most important aspect of this. We wanted to link every single transaction to mock data and process that in our primary system.<\/p>\n\n\n\n<p>To facilitate this we decided to move forward in modelling our current infrastructure on production, but taken to a level that allows for increased scale.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Understanding our infrastructure<\/strong><\/h2>\n\n\n\n<p>We use <a href=\"https:\/\/cloud.google.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">Google Cloud Services<\/a> (GCS) for our hosted database, as well as smaller features of our ecosystem like serverless functions.<\/p>\n\n\n\n<p>For our main backend application, written in <a href=\"https:\/\/laravel.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">Laravel<\/a> and Hedera API servers, we utilise Digital Ocean.&nbsp;<\/p>\n\n\n\n<p><a href=\"https:\/\/forge.laravel.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">Laravel Forge<\/a> manages the infrastructure for our Laravel application.<\/p>\n\n\n\n<p>Our Hedera REST API, Trust Enterprises, uses the <a href=\"https:\/\/www.digitalocean.com\/products\/app-platform\" target=\"_blank\" rel=\"noreferrer noopener\">App Platform<\/a>. We could save money on this side of things but, we try to opt for simplicity and ease of use.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Driving Hedera transactions through an API, a note<\/strong><\/h2>\n\n\n\n<p>Since working on Hedera, there has always been one item that has differentiated it from other platforms. This is the accessibility for any developer to build upon the network. <a href=\"https:\/\/hedera.com\/consensus-service\" target=\"_blank\" rel=\"noreferrer noopener\">Consensus<\/a> or <a href=\"https:\/\/hedera.com\/token-service\" target=\"_blank\" rel=\"noreferrer noopener\">Token Service<\/a> are easy to work with, even if you have limited knowledge of JavaScript or other languages. <\/p>\n\n\n\n<p>I developed Trust Enterprises with the view that any developer should be able to access these services through a REST API with a client node that is simple to deploy, meaning that regardless of any specific language or infrastructure it would be easy to start working with the system.<\/p>\n\n\n\n<p><strong>However, due to the nature of using an API and HTTP request timings, there will always be a bottleneck of throughput.<\/strong><\/p>\n\n\n\n<p>To combat this, in our core application we take advantage of the job queueing system that is present in Laravel. Every time we send a transaction to Hedera we utilise jobs as a means for parallelising our throughput. Because of this, we can&#8217;t have sudden bursts of high TPS.<\/p>\n\n\n\n<p>However, the problem with this is that you need to make sure that the API that handles connections to the Hedera network is capable of dealing with large amounts of traffic, especially given that particular nodes frequently throttle connections in testnet conditions.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What causes throttling on testnet?<\/h2>\n\n\n\n<p>This throttling on testnet has been a particular ecosystem-wide issue. In many cases, when a Hedera node is having issues processing transactions, it can trigger a cascade effect of the underlying SDK becoming \u201cstuck\u201d. At this point, there&#8217;s seemingly no obvious way to pivot to another node to send transactions to.<\/p>\n\n\n\n<p>With this in mind, there are a couple of issues, you have to be aware of:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>That, in testnet conditions, you might hit throttling issues if you rapidly tried to push more than 30-40 TPS consistently.<\/li><li>In some cases, one has to reset a given infrastructure in order for transactions to be processed once again. <\/li><\/ul>\n\n\n\n<p>We quickly learned that having a single large server to process all transactions from our core API to Hedera was prohibitive. It triggered these bottlenecks more frequently than we would have liked.&nbsp;<\/p>\n\n\n\n<p>Thus, in order to drive transaction volume linked to meaningful data, the approach is to horizontally scale systems, rather than vertically. <\/p>\n\n\n\n<p>Vertical scaling in this case is a way to increase throughput by increasing the power of your server directly.<\/p>\n\n\n\n<p>Horizontally scaling, on the other hand, relies on the concepts of mitigating redundancy and having multiple lower-powered nodes.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Breaking down the costs<\/strong><\/h2>\n\n\n\n<p>For both primary infrastructure elements, we chose to duplicate resources that were already present in our production system. For this round we named our infrastructure with the <strong>stardestroyer <\/strong>prefix, to unleash the beast.<\/p>\n\n\n\n<p>Comprising the two elements, below is the breakdown of the costs, specifications, and additional details.&nbsp;<\/p>\n\n\n\n<p>We took the view that if we needed to spend approximately $1,200 a month on infrastructure demands, reasonable in the medium term, we would provision infrastructure to match. However, during the course of the test, we wanted to run for approximately five days so the actual cost of the test would be around the $200 mark.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Core Laravel App<\/strong><\/h2>\n\n\n\n<p>It was vital that we used dedicated CPUs with a high level of memory. This was due to the nature of utilising Laravel\u2019s Job queue mechanism with a number of different workers.<\/p>\n\n\n\n<p>We decided on the specification for approximately $640 a month, excluding tax.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh4.googleusercontent.com\/VtC0HCjpvab3DrCZTVKgxJrVQwZnNsAGz9kPTZ8TZOeTClYLJMpOtJZWc6zjh10tomiLpe4n3mI_3lmH1c5Be3j0MQltEPIxeo-qlMJRmlDA-bdoifSuUGVUjzlKBy5nsl7P6Syb\" alt=\"\"\/><figcaption>Memory Optimized Droplets<\/figcaption><\/figure>\n\n\n\n<p>In this case, we just had a single node that our main application ran on to make it simple. We could further extend this by having a load balancer with multiple backend nodes and separating out databases for distributing different read and write functions. This, I believe, would be overkill for this test. If we needed that infrastructure demand, it would be within the view of the long-term &#8211; after two or three years.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Trust Enterprises Hedera API<\/strong><\/h2>\n\n\n\n<p>We uncovered a lot about Hedera through provisioning our API over the course of the week. Initially, we decided to use the single node, to keep things simple. We opted to use <a href=\"https:\/\/try.digitalocean.com\/app-platform\" target=\"_blank\" rel=\"noreferrer noopener\">Digital Ocean\u2019s app platform<\/a> as it enabled us to connect a GitHub repository to a system that would detect a given language, and then deploy through <a href=\"https:\/\/www.docker.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">Docker<\/a> an automated process.<\/p>\n\n\n\n<p>We opted for an initial container of the largest spec the platform had to offer. Why? Because on our staging infrastructure we utilise a single container to keep costs down. For this round, we decided to increase the node size\u2026 at least to begin with.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh3.googleusercontent.com\/yYE7JQoDyU09rsMYuKh0WQJG_YlnSntM8TglMm3vsEtusj5wdJKJUEgm0_bd4M2NGe7OAUophqT_NqBg25kbFah63ejmvoqsdbujRvVf7-WTYVlUb4ToOopL2rXcif7ldVUCwz_f\" alt=\"\"\/><figcaption>Containers<\/figcaption><\/figure>\n\n\n\n<p>Obviously, we decided this when we planned to use a single container for a vertically scaled system. When we pivoted to a horizontally scaled approach, we switched to having lots of small containers to deal with a much lower volume of transactions per node.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh4.googleusercontent.com\/31S9tQNabvJKXEVeljgPkCPimoxK8REvXBaGFkS1x4eRqTUnSk5htsyoB5XdvvTjSFnkXpzTwyD4J0yfWII1wU6OEfMh_oDBAf5E_c-CQ24YZ4nJclp10pkoHBO0s348LdVaUQy3\" alt=\"\"\/><figcaption>Component Settings<\/figcaption><\/figure>\n\n\n\n<p>From a cost perspective, nothing changed. But we ended up building our infrastructure up to 25 containers, to spread the transaction load between them all. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Understanding DOVU\u2019s Staking platform<\/strong><\/h2>\n\n\n\n<p>There are a number of things that happened throughout this test. At first, we decided to slowly build up the users in our staging environment. This ensured that we could handle the expected throughput we were aiming for. As part of our APIs and tooling, we have tools to generate users and to send tokens. For a test environment you have to complete a few steps:<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>Generate a Hedera account<\/li><li>Associate the account with a given token ID, which reflects a mock token for staking (tDOV)<\/li><li>Send a small amount of that token to the account<\/li><\/ol>\n\n\n\n<p>DOVU\u2019s staking platform relies on a key principle of Hedera, that is:&nbsp;<\/p>\n\n\n\n<p><strong><em>Tokens must be associated to an account, if an account holds a token it must have been associated.<\/em><\/strong><\/p>\n\n\n\n<p>From here we can infer that, if a user has a particular balance, we can take that as gospel from the network and calculate the share of a given reward for a particular hour.<\/p>\n\n\n\n<p>As part of the initial test, we generated over 20,000 accounts. Our goal was to send a tDOV to all users every 1-2 minutes, so we could reach our target TPS.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh5.googleusercontent.com\/IVpqzV4eGOEptDdFhQUQDqeIKm1JQWkrsYivezLmQ0MW0e0rIDP6j1u4z678DSnDYD1fXa49UmH2cVToQ0otQXwwj0jZcV9tJf2S8qW-yxyeQn1U-1V8M8Nl6n7VrC43nQsOJqZf\" alt=\"\"\/><figcaption>New users<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Ramping up<\/strong><\/h2>\n\n\n\n<p>Once we drilled in our infrastructure and we were gaining reasonable consistency for sending the tokens every two minutes, we kept an eye on it.&nbsp;<\/p>\n\n\n\n<p>In order to drive these transactions we used <a href=\"https:\/\/laravel.com\/docs\/9.x\/horizon\" target=\"_blank\" rel=\"noreferrer noopener\">Laravel Horizon<\/a>, the core system for managing our Redis Queues. It provided us with a helpful panel and gave us all the necessary details. This way we could keep on top of any failing jobs, and adjust our infrastructure accordingly.<\/p>\n\n\n\n<p>At our peak, we were pushing through just under 5000 jobs a minute. This comprised a maximum of 500 processes. You could effectively consider these threads for concurrency.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh5.googleusercontent.com\/FRg_kpZ1Aj9MVdU6f4WwR47pwHyquGxC9WVrS9VVxqFoWQd95MI5knh6FcPqb-RAQWDDWPzopW4OFnGbhzyF9dLO5ne5ZYivl70PQ8PEFvz0G-ChvbjgQxuR9CsDGQPUkGmEoHIX\" alt=\"\"\/><figcaption>DOVU Carbon Marketplace on Laravel Horizon<\/figcaption><\/figure>\n\n\n\n<p>Between our queue dashboard, <a href=\"https:\/\/app.dragonglass.me\/hedera\/home\" target=\"_blank\" rel=\"noreferrer noopener\">Dragonglass<\/a>, and <a href=\"https:\/\/hederatxns.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">HederaTxns<\/a> we had a good view of the analytics needed to diagnose any issues during the stress test.&nbsp;<\/p>\n\n\n\n<p>Interestingly enough, the amount of jobs in a queue became a good indicator of the current throughput we could push through Hedera. When the network was failing to process around 200-250 TPS the actual jobs waiting to be processed would increase on the \u201cCurrent Workload\u201d panel.<\/p>\n\n\n\n<p>Part of our core philosophy at DOVU is to have redundancy at the heart of our systems. This means, as part of our admit panel, if tasks fail we have fine-grained control to retry a given task. Task failures include if tokens fail to be transferred, or other jobs faily to complete. In the images below you can see an example of \u201cRetry Token Transfer\u201d.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Hedera testnet throttling<\/strong><\/h2>\n\n\n\n<p>Within the ecosystem there have been complaints around how particular SDKs handle the selection of nodes when sending messages to them. Generally, when you try to push too many transactions to a particular node too quickly in a given SDK, it will use the Javascript one. It can be a challenge to recover from this, to bypass and to reattempt the transaction to another node.&nbsp;<\/p>\n\n\n\n<p>It&#8217;s almost like a given container is unable to retroactively step back and choose a different node to send a transaction to.<\/p>\n\n\n\n<p>I&#8217;ve been aware of this issue for some time, and it happened once during the test. It\u2019s really hard to illustrate but here&#8217;s an example of the issue:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh6.googleusercontent.com\/jcpPUrG3_oO0OtC0YzOY8ocKbN7slpEDNfJimeQC5C0Dd25_i5AWVD3CwgMeABhI_E2COibthhFF1T513K4Y0DkJ1omJHSVlCjDw2RlESiF10Vh-K3tr-auHPe9lP77I4bKz-B82\" alt=\"\"\/><figcaption>Hedera testnet throttling<\/figcaption><\/figure>\n\n\n\n<p>This is a screenshot of our Hedera API continually processing transactions, over a six hour period. At approximately 1145 we have a dip of transaction throughput. After that point, only a fraction of the containers are able to process current transactions to the system. It stands to reason that a number of containers did get stuck; we needed to reset the app to get them to start running again.<\/p>\n\n\n\n<p>It is worth noting that we were public about this stress test. During the day, there was another mysterious actor that was pushing 5,000-8,000 transactions through the system periodically.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Enter: the enigmatic step function<\/strong><\/h2>\n\n\n\n<p>It seems that the more throughput there is going through Hedera, the higher the chance of throttling testnet. We need to diagnose and remedy whether this is an issue with DOVU\u2019s system or with the network itself. But even in periods of congestion, we still managed to maintain 100 TPS.<\/p>\n\n\n\n<p>Every single one of our transactions represented some state within our system; primarily staking payments at scale.<\/p>\n\n\n\n<figure class=\"wp-block-image is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/lh4.googleusercontent.com\/N3aS4xWgIBYishqETUzSTqFOdVBVMuV87-baCgxCRXBdsF7LjNwPLow8CFRQF8Mk_wi0TMeJYxiyaNEJBQs2PTFWPE6c2aNSF5hSZKcKa8wlqtElqhefYpUFgeuY2AnpvBptl2eN\" alt=\"\" width=\"840\" height=\"395\"\/><figcaption>Staking payments<\/figcaption><\/figure>\n\n\n\n<p>Naturally, you can see by the reward that the value differs for every single wallet. This means we can calculate the exact amount of value (to 6 decimal places) to send to a given user.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Outcome of the test<\/strong><\/h2>\n\n\n\n<p>It\u2019s unlikely that we&#8217;ll hit 250+ TPS in the near future. But this performance test was absolutely vital for our learnings, including:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>What we need to do to expand<\/li><li>If there are there easy optimisations at play&nbsp;<\/li><li>If additional infrastructure is required to be provisioned<\/li><\/ul>\n\n\n\n<p>There are still a few congestion issues surrounding consistently driving anything above 200 TPS without having to reset an app. At this point, it\u2019s unknown whether this is a testnet issue, or if it would affect mainnet as well.<\/p>\n\n\n\n<p>We were impressed with the amount of throughput we pushed to the network during the test &#8211; hitting over 20% more than our estimations. You can slowly scale up and meet your own internal system&#8217;s demands, provided you have enough infrastructure and budget.&nbsp;<\/p>\n\n\n\n<p>One process challenge is that we currently use an API to send transactions to Hedera. If we use a native gRPC library, using the JS SDK on a nodeJS app, this would remove a bottleneck for maximising the use of the network. We use Laravel as it provides so much out of the box for us as the foundation to develop against. And we can bolt on additional microservices as required.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>What can you do to drive this TPS yourself?<\/strong><\/h2>\n\n\n\n<p>Our entire Hedera API is based on the <a href=\"https:\/\/github.com\/trustenterprises\/hedera-serverless-api\" target=\"_blank\" rel=\"noreferrer noopener\">Trust Enterprises API<\/a>, which is an open source project. It can be forked and configured for any Hedera SDK method for your individual needs. It\u2019s fully tested and comes with CI\/CD tooling by default.&nbsp;<\/p>\n\n\n\n<p>I implore you to follow this guide and to take a look at Digital Ocean. I particularly recommend looking at the <a href=\"https:\/\/www.digitalocean.com\/products\/app-platform\" target=\"_blank\" rel=\"noreferrer noopener\">App Platform<\/a> and <a href=\"https:\/\/docs.digitalocean.com\/products\/droplets\/\" target=\"_blank\" rel=\"noreferrer noopener\">Droplets<\/a>.&nbsp;<\/p>\n\n\n\n<p>It\u2019s possible to mirror what we have done starting from less than $80 a month. As a starting point, this could be your initial specification for your Hedera-powered infrastructure.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>A droplet that costs $20-$40 a month for the core production app<\/li><li>Combined with the app engine with 2 containers for $12 each, to make $24 dollars<\/li><\/ul>\n\n\n\n<p>From here you can continually test the demands of your system. But you should be able to consistently push 40-50 TPS to the Hedera testnet from this starting point.<\/p>\n\n\n\n<p>If you\u2019ve found this useful &#8211; and especially if you\u2019ve built something using some of these insights &#8211; get in touch. I\u2019d love to hear how you get on.<\/p>\n\n\n\n<p>Join our <a href=\"https:\/\/discord.gg\/fwgWrtZn\" target=\"_blank\" rel=\"noreferrer noopener\">Discord<\/a> and find me there!<\/p>\n\n\n\n<hr class=\"wp-block-separator is-style-dots\"\/>\n\n\n\n<ul class=\"wp-block-list\"><li>Be the first in the know:&nbsp;<a href=\"https:\/\/discord.gg\/wW7rg6dDN4\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/discord.gg\/wW7rg6dDN4<\/a><\/li><li>Browse:&nbsp;<a href=\"https:\/\/dovu.earth\/en\/\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/dovu.earth\/en\/<\/a><\/li><li>Connect:&nbsp;<a href=\"https:\/\/twitter.com\/dovuofficial\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/twitter.com\/dovuofficial<\/a><\/li><li>Read:&nbsp;<a href=\"https:\/\/medium.com\/dovu-earth\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/medium.com\/dovu-earth<\/a><\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"<p>DOVU CTO Matt Smithies writes about our recent public performance test and shares insights for others looking to run similar experiments. In Q1 2022 we ran a public performance test on the Hedera Hashgraph testnet. We wanted to prove that we are able to scale in line with a vision.&nbsp;&#8230;<\/p>\n<div class=\"mt-4\"><a class=\"read-more ajax-blogpost\" href=\"https:\/\/dovu.earth\/en\/scaling-infrastructure-how-we-did-it\/\" data-id=\"1520\">Read More<\/a><\/div>\n","protected":false},"author":2,"featured_media":1401,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[4],"tags":[18,19,22,78],"class_list":["post-1520","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news-insight","tag-dovu","tag-dovu-product","tag-hedera","tag-testnet"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Scaling Infrastructure: How we did it - DOVU<\/title>\n<meta name=\"description\" content=\"We were impressed with the amount of throughput we pushed to the network during the test - hitting over 20% more than our estimations.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/dovu.earth\/en\/scaling-infrastructure-how-we-did-it\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Scaling Infrastructure: How we did it - DOVU\" \/>\n<meta property=\"og:description\" content=\"We were impressed with the amount of throughput we pushed to the network during the test - hitting over 20% more than our estimations.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/dovu.earth\/en\/scaling-infrastructure-how-we-did-it\/\" \/>\n<meta property=\"og:site_name\" content=\"DOVU\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DOVU.Earth\/\" \/>\n<meta property=\"article:published_time\" content=\"2022-03-30T12:57:21+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-03-30T12:57:23+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/dovu.earth\/wp-content\/uploads\/2021\/10\/DOVU_solution_carbon_market.png\" \/>\n\t<meta property=\"og:image:width\" content=\"630\" \/>\n\t<meta property=\"og:image:height\" content=\"330\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Dovu Admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@dovuofficial\" \/>\n<meta name=\"twitter:site\" content=\"@dovuofficial\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Dovu Admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"15 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/dovu.earth\\\/scaling-infrastructure-how-we-did-it\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/dovu.earth\\\/scaling-infrastructure-how-we-did-it\\\/\"},\"author\":{\"name\":\"Dovu Admin\",\"@id\":\"https:\\\/\\\/dovu.earth\\\/#\\\/schema\\\/person\\\/ab9d15c5797c1090f9d0a8e570daf4c2\"},\"headline\":\"Scaling Infrastructure: How we did it\",\"datePublished\":\"2022-03-30T12:57:21+00:00\",\"dateModified\":\"2022-03-30T12:57:23+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/dovu.earth\\\/scaling-infrastructure-how-we-did-it\\\/\"},\"wordCount\":2791,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/dovu.earth\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/dovu.earth\\\/scaling-infrastructure-how-we-did-it\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/dovu.earth\\\/wp-content\\\/uploads\\\/2021\\\/10\\\/DOVU_solution_carbon_market.png\",\"keywords\":[\"DOVU\",\"DOVU Product\",\"Hedera\",\"testnet\"],\"articleSection\":[\"BLOG POSTS\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/dovu.earth\\\/scaling-infrastructure-how-we-did-it\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/dovu.earth\\\/scaling-infrastructure-how-we-did-it\\\/\",\"url\":\"https:\\\/\\\/dovu.earth\\\/scaling-infrastructure-how-we-did-it\\\/\",\"name\":\"Scaling Infrastructure: How we did it - DOVU\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/dovu.earth\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/dovu.earth\\\/scaling-infrastructure-how-we-did-it\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/dovu.earth\\\/scaling-infrastructure-how-we-did-it\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/dovu.earth\\\/wp-content\\\/uploads\\\/2021\\\/10\\\/DOVU_solution_carbon_market.png\",\"datePublished\":\"2022-03-30T12:57:21+00:00\",\"dateModified\":\"2022-03-30T12:57:23+00:00\",\"description\":\"We were impressed with the amount of throughput we pushed to the network during the test - hitting over 20% more than our estimations.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/dovu.earth\\\/scaling-infrastructure-how-we-did-it\\\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/dovu.earth\\\/scaling-infrastructure-how-we-did-it\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\\\/\\\/dovu.earth\\\/scaling-infrastructure-how-we-did-it\\\/#primaryimage\",\"url\":\"https:\\\/\\\/dovu.earth\\\/wp-content\\\/uploads\\\/2021\\\/10\\\/DOVU_solution_carbon_market.png\",\"contentUrl\":\"https:\\\/\\\/dovu.earth\\\/wp-content\\\/uploads\\\/2021\\\/10\\\/DOVU_solution_carbon_market.png\",\"width\":630,\"height\":330},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/dovu.earth\\\/scaling-infrastructure-how-we-did-it\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/dovu.earth\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Scaling Infrastructure: How we did it\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/dovu.earth\\\/#website\",\"url\":\"https:\\\/\\\/dovu.earth\\\/\",\"name\":\"DOVU\",\"description\":\"Home\",\"publisher\":{\"@id\":\"https:\\\/\\\/dovu.earth\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/dovu.earth\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/dovu.earth\\\/#organization\",\"name\":\"DOVU\",\"url\":\"https:\\\/\\\/dovu.earth\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\\\/\\\/dovu.earth\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/dovu.earth\\\/wp-content\\\/uploads\\\/2021\\\/06\\\/Dovu-Logo-Icon.svg\",\"contentUrl\":\"https:\\\/\\\/dovu.earth\\\/wp-content\\\/uploads\\\/2021\\\/06\\\/Dovu-Logo-Icon.svg\",\"width\":97,\"height\":97,\"caption\":\"DOVU\"},\"image\":{\"@id\":\"https:\\\/\\\/dovu.earth\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/DOVU.Earth\\\/\",\"https:\\\/\\\/x.com\\\/dovuofficial\",\"https:\\\/\\\/www.instagram.com\\\/dovuofficial\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/11116224\\\/admin\\\/\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UCxiQtFv5gIaxIr1hnF1PQOw\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/dovu.earth\\\/#\\\/schema\\\/person\\\/ab9d15c5797c1090f9d0a8e570daf4c2\",\"name\":\"Dovu Admin\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Scaling Infrastructure: How we did it - DOVU","description":"We were impressed with the amount of throughput we pushed to the network during the test - hitting over 20% more than our estimations.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/dovu.earth\/en\/scaling-infrastructure-how-we-did-it\/","og_locale":"en_GB","og_type":"article","og_title":"Scaling Infrastructure: How we did it - DOVU","og_description":"We were impressed with the amount of throughput we pushed to the network during the test - hitting over 20% more than our estimations.","og_url":"https:\/\/dovu.earth\/en\/scaling-infrastructure-how-we-did-it\/","og_site_name":"DOVU","article_publisher":"https:\/\/www.facebook.com\/DOVU.Earth\/","article_published_time":"2022-03-30T12:57:21+00:00","article_modified_time":"2022-03-30T12:57:23+00:00","og_image":[{"width":630,"height":330,"url":"https:\/\/dovu.earth\/wp-content\/uploads\/2021\/10\/DOVU_solution_carbon_market.png","type":"image\/png"}],"author":"Dovu Admin","twitter_card":"summary_large_image","twitter_creator":"@dovuofficial","twitter_site":"@dovuofficial","twitter_misc":{"Written by":"Dovu Admin","Estimated reading time":"15 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/dovu.earth\/scaling-infrastructure-how-we-did-it\/#article","isPartOf":{"@id":"https:\/\/dovu.earth\/scaling-infrastructure-how-we-did-it\/"},"author":{"name":"Dovu Admin","@id":"https:\/\/dovu.earth\/#\/schema\/person\/ab9d15c5797c1090f9d0a8e570daf4c2"},"headline":"Scaling Infrastructure: How we did it","datePublished":"2022-03-30T12:57:21+00:00","dateModified":"2022-03-30T12:57:23+00:00","mainEntityOfPage":{"@id":"https:\/\/dovu.earth\/scaling-infrastructure-how-we-did-it\/"},"wordCount":2791,"commentCount":0,"publisher":{"@id":"https:\/\/dovu.earth\/#organization"},"image":{"@id":"https:\/\/dovu.earth\/scaling-infrastructure-how-we-did-it\/#primaryimage"},"thumbnailUrl":"https:\/\/dovu.earth\/wp-content\/uploads\/2021\/10\/DOVU_solution_carbon_market.png","keywords":["DOVU","DOVU Product","Hedera","testnet"],"articleSection":["BLOG POSTS"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/dovu.earth\/scaling-infrastructure-how-we-did-it\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/dovu.earth\/scaling-infrastructure-how-we-did-it\/","url":"https:\/\/dovu.earth\/scaling-infrastructure-how-we-did-it\/","name":"Scaling Infrastructure: How we did it - DOVU","isPartOf":{"@id":"https:\/\/dovu.earth\/#website"},"primaryImageOfPage":{"@id":"https:\/\/dovu.earth\/scaling-infrastructure-how-we-did-it\/#primaryimage"},"image":{"@id":"https:\/\/dovu.earth\/scaling-infrastructure-how-we-did-it\/#primaryimage"},"thumbnailUrl":"https:\/\/dovu.earth\/wp-content\/uploads\/2021\/10\/DOVU_solution_carbon_market.png","datePublished":"2022-03-30T12:57:21+00:00","dateModified":"2022-03-30T12:57:23+00:00","description":"We were impressed with the amount of throughput we pushed to the network during the test - hitting over 20% more than our estimations.","breadcrumb":{"@id":"https:\/\/dovu.earth\/scaling-infrastructure-how-we-did-it\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/dovu.earth\/scaling-infrastructure-how-we-did-it\/"]}]},{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/dovu.earth\/scaling-infrastructure-how-we-did-it\/#primaryimage","url":"https:\/\/dovu.earth\/wp-content\/uploads\/2021\/10\/DOVU_solution_carbon_market.png","contentUrl":"https:\/\/dovu.earth\/wp-content\/uploads\/2021\/10\/DOVU_solution_carbon_market.png","width":630,"height":330},{"@type":"BreadcrumbList","@id":"https:\/\/dovu.earth\/scaling-infrastructure-how-we-did-it\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/dovu.earth\/"},{"@type":"ListItem","position":2,"name":"Scaling Infrastructure: How we did it"}]},{"@type":"WebSite","@id":"https:\/\/dovu.earth\/#website","url":"https:\/\/dovu.earth\/","name":"DOVU","description":"Home","publisher":{"@id":"https:\/\/dovu.earth\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/dovu.earth\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/dovu.earth\/#organization","name":"DOVU","url":"https:\/\/dovu.earth\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/dovu.earth\/#\/schema\/logo\/image\/","url":"https:\/\/dovu.earth\/wp-content\/uploads\/2021\/06\/Dovu-Logo-Icon.svg","contentUrl":"https:\/\/dovu.earth\/wp-content\/uploads\/2021\/06\/Dovu-Logo-Icon.svg","width":97,"height":97,"caption":"DOVU"},"image":{"@id":"https:\/\/dovu.earth\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DOVU.Earth\/","https:\/\/x.com\/dovuofficial","https:\/\/www.instagram.com\/dovuofficial\/","https:\/\/www.linkedin.com\/company\/11116224\/admin\/","https:\/\/www.youtube.com\/channel\/UCxiQtFv5gIaxIr1hnF1PQOw"]},{"@type":"Person","@id":"https:\/\/dovu.earth\/#\/schema\/person\/ab9d15c5797c1090f9d0a8e570daf4c2","name":"Dovu Admin"}]}},"_links":{"self":[{"href":"https:\/\/dovu.earth\/en\/wp-json\/wp\/v2\/posts\/1520","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dovu.earth\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dovu.earth\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dovu.earth\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/dovu.earth\/en\/wp-json\/wp\/v2\/comments?post=1520"}],"version-history":[{"count":0,"href":"https:\/\/dovu.earth\/en\/wp-json\/wp\/v2\/posts\/1520\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dovu.earth\/en\/wp-json\/wp\/v2\/media\/1401"}],"wp:attachment":[{"href":"https:\/\/dovu.earth\/en\/wp-json\/wp\/v2\/media?parent=1520"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dovu.earth\/en\/wp-json\/wp\/v2\/categories?post=1520"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dovu.earth\/en\/wp-json\/wp\/v2\/tags?post=1520"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}