To create a sound opinion of someones service is a serious and complex task. Throughout the last decade we ran an uncounted number of consulting engagements where we supported our customers in selecting all kinds of enterprise software.
Across all these projects, we have learned that almost all cases saw a segregation of several dimensions: there were business aspects, service aspects and product specific aspects. The more sophisticated the buyer, the more importance gained the sustainability aspect.
Based on this experience, we have identified a structure, which we think is a great approach to derive a sound opinion on an IaaS offering. The structure is depicted in the following diagram.
Structure of IaaS Rating Tree
The first level is represented by four dimensions: Technical Performance, Technical Capabilities, Service Performance and Provider "Restainability". The latter stands for a mixture of REliability and suSTAINABILITY aspects. For our General Purpose Rating we decided the weighting as depicted in the diagram. The relevance of the different dimensions might change for a particular use case or in an individual approach. Feel free to contact us, in case you want an individual rating to be exercised.
The first aspect directly addresses the technical performance. Here we take data from our measurements and compare the expected quality against the market average. Based on the performance of the indicators CPU (compute capacity, ASC), Disk I/O (Mb/s read/write file, DB and log) and Network throughput & latency (Mb/s) - see individual tests for more details - in relation to the market average a valuation will be derived.
This also is one of the reasons, why ratings will change over time. The more providers are added to the database, the average will change. This could shift providers in a new position, even if they did not change anything. However, we expect this effect to disappear as maturity increases.
Besides the individual performance of the virtual machines or containers, which is in the end a configuration issue, the technical equipment, the architecture and other aspects setup in the strategic alignment of the provider determine a certain quality. If the datacenters used are not even Uptime level 3, it will be difficult to provide a reliable service. Therefor we assess the capabilities, the things behind the scene to get a sound understanding of the capabilities the provider has and can offer.
The capabilities also determine the midterm options a provider has and therefor got the heaviest weight in the overall rating.
In a corporate context, especially when you plan to run mission critical applications, service and service performance also play a major role. In the beginning of 2016 a smaller American provider suffered from a major outage due to a DDoS attack. All representatives and relevant people were on holiday, leaving nobody to re-establish operations for over ten business days.
To prevent you from experiencing something like this, this dimension looks at SLA terms, guarantees given, refund mechanisms, internal procedures and policies, etc.
Finally you do not want to decide for a partner who today makes a great look but does bear the risk of failing tomorrow. Therefor we conduct a typical financial assessment, try to find out about strategy, current and future market positioning, derive based on interviews with management and financial figures an understanding on where we expect the company to be in three or five years from now.
This also includes R&D spending, partnerships maintained, technologies applied, references and successful projects.
Each of the criteria has a value span between 0 and 5. Based on the weights the criteria are calculated to sum up to the general rating. See "Understand Ratings" for more details on the interpretation of our rating information.