FinOps , or Financial Operations, is defined as “a framework and cultural practice focused on bringing financial responsibility to cloud spending in a way that balances cost, operational efficiency, and quality.” FinOps enables cross-functional teams, such as finance, engineering, product, and operations, to collaborate to effectively manage cloud costs and maximize business value. As a byproduct, the concept also includes a sustainability perspective. Note! FinOps can also be done on-premise in connection with data migrations. Data migrations are typically large-scale, multi-disciplinary projects that have a significant impact on business. Data migration is the process of selecting, preparing, extracting, and possibly transforming data and permanently moving it from one data environment to another. Data migration of SQL Server platforms can occur for several reasons, such as changing servers or storage devices, consolidation, and moving a data center from on-prem to a public cloud, such as Microsoft Azure. Cost reasons and their transparency, the need for the database to be close to applications (cloud-migrated applications), data security, and operational efficiency are key drivers when moving to Azure. However, the costs of the data platform are not always automatically more advantageous in the public cloud than on-prem, especially if the FinOps component is overlooked. SQL Server data migrations are usually performed in a way that is as automated as possible, which frees up human resources from repetitive routines, and to avoid errors that easily occur in manual repetitive work. Although migrations are often highly automated, they must be monitored and the success of the migrations must always be carefully verified: “What goes out, must come in”. FinOps areas FinOps can be divided into five different areas:
- Visibility and responsibility
- Cost optimization
- Operational efficiency
- Effective teams
- Data-driven decision-making
Visibility and accountability refer to real-time transparency of costs, predictability, and control between different stakeholders. of cost optimization is to achieve savings and better TCO without compromising service performance or availability. This includes concepts such as right-sizing, consolidation, and demand-driven dynamic scaling, which are some of the benefits of a public cloud like Azure. Operational efficiency means that cost monitoring is continuous and comprehensive, allowing for rapid response to changing needs. Effective teams mean that data engineers, when designing and further developing the data platform, understand and manage the impact of architecture and configurations on the costs of the data platform. This contributes to sharing responsibility for the overall costs. Data-driven decision-making with FinOps is based on reliable data, which helps organizations better predict and manage their costs. The role of FinOps in data migrations FinOps plays a significant role in data migrations, especially as organizations move workloads and data from on-premises environments to the cloud. Given the cost, complexity, and financial impact of cloud migrations, FinOps can provide a framework for effectively managing and optimizing these costs. FinOps is divided into the following areas in data migrations:
- Pre-migration planning and cost estimation
- Cost optimization during migration
- Post-migration capacity and cost optimization
- Governance and accountability
- Data-driven insights and reporting
Pre-migration planning and cost estimation Pre-migration planning and cost estimation are divided into the following subsections:
- SQL Server performance data monitoring
- Budgeting, forecasting and capacity planning
- Setting goals (KPI)
SQL Server performance data should be collected from all SQL Servers, instances, and databases so that capacity planning and consolidation can be done comprehensively at all levels of the data platform. Suitable products for this are tools specialized in monitoring and capacity planning for SQL Servers, such as SQL Governor , which can also identify unused database resources early in the project so that they are not unnecessarily moved to a new environment, generating additional costs. Budgeting and forecasting are best done by combining empirical knowledge from previous similar migrations and assessing capacity needs by calculating different scenarios for logical areas, such as development, test, and production environments by business area. The following methods can be applied to or related to server clusters, servers, instances in capacity planning :
- Lift and Shift (moving workloads to the new environment “as is”)
- Right-sizing, i.e. resizing servers or Managed instances
- Server-level consolidation (optimizing the total number of hosts)
- Instance-level consolidation (optimizing the number of hosted virtual machines, Managed Instance pools, or physical servers)
- Database-level consolidation (harmonizing workloads down to the instance level)
When setting goals, the budget and financial KPIs are defined, within which the data migration must be fully implemented and the desired ROI achieved. Cost optimization during migration FinOps teams monitor migration costs in real time, and identify any unexpected costs or resource spikes to keep costs under control. FinOps best practices and capacity planning software such as SQL Governor help to significantly adjust resource usage based on historical forecasts and real-time needs, avoiding overprovisioning of services. However, it may be that, for example, with the purchase of a version of SQL Server, especially if the previous version is already old (2008R2 – 2016), the new service platform will require optimization of the new database instance, such as adjusting instance and database level configurations, such as Max degree of parallelism and Cost threshold for parallelism settings. Other optimizations should be left until after the migration, when telemetry and SQL Server usage statistics have been collected from the new environment. However, Query Store should be turned on immediately after the migration. Cloud platforms also offer various savings plans and commitments for long-term migrations, allowing for additional savings. Post-migration capacity and cost optimization After the migration, the FinOps team ensures that costs are accurately allocated to different departments or projects so that each team understands their own costs. The FinOps team also helps identify any unused resources that may be left behind and facilitates their removal to avoid unnecessary costs. The FinOps team continuously strives to optimize resources in the new cloud environment, ensuring that workloads remain cost-effective over time. Usually, for the first time after 1 full month (=business cycle) of migration, it is extremely important to benchmark the performance of the new database environment at the server, instance and database level compared to the old environment. For example, the SQL Governor product has a benchmarking tool that uses KPIs and graphs to easily see the bottlenecks and progress of the system. If optimization needs arise, the SQL platform can be optimized, for example by forcing better performing execution plans on the database instance from the Query Store , or even by using the Legacy Cardinality Estimator for problem databases. It is also not uncommon to sometimes have to rewrite some T-SQL code or tweak indexing, if this is possible from the application vendor’s side. These problems often become apparent gradually in already migrated environments, so some of these measures, especially those related to indexing and code optimization, can only be taken after migration. SQL Governor also offers a good collection of tools for resolving many of these problems. Governance and accountability During migration, the FinOps team can establish policies, such as guidelines on acceptable spending, resource consumption, and data exit restrictions, to ensure cost control. Furthermore; by providing real-time visibility into migration costs, FinOps empowers planning teams to make cost-informed choices and be accountable for their spending. Data-driven insights and reporting FinOps provides regular reports and dashboards that help stakeholders understand migration costs, track progress against budget, and ensure the organization’s financial efficiency. For example, in Azure, billing telemetry and forecasts for existing configurations are available “out of the box”. This, combined with the ability of our AI-based SQL Governor product to calculate capacity needs and consolidation opportunities for existing databases, instances, and virtual machines in different configurations over the long term, creates a truly powerful combination that can be used to optimize TCO in the best possible way. Lift and Shift – poison for TCO A word of caution regarding “Lift and Shift”, it is not all bedrock. The larger the database environment, the more things inevitably go wrong with this thinking: A large proportion of databases are over- or under-scaled: Over-scaled databases do not perform well enough, and therefore generate unavailability situations and hidden costs. Under-scaled databases, on the other hand, become unnecessarily expensive, reserving extra capacity, especially in Azure VMs and Managed Instances, and Azure SQL Hyperscale does not come for free. The idea behind Lift and Shift is good, minimizing visible migration costs and time-to-solution, but it can cause a huge headache, for example when moving to a public cloud like Microsoft Azure. Although moving to Azure with Lift and Shift may sound easy at first glance, in practice it is often not: When you have to change storage capacity service tiers, guess the size or type of a virtual machine from one to another, or encounter a higher availability configuration that cannot be downgraded afterwards, you are in trouble – this information must also be based on predictive telemetry data so that the solution is on a sustainable basis. Just staring at the transom is not enough. In addition, there are unavailability situations and downtime caused by re-optimization / reconfiguration. In this case, money and time are returned while the SLAs of the services are reduced. Of course, servers, instances and databases can also be optimized after migration, but this process takes just as much time as before migration, which also makes the total cost of investment a big question mark. After the service is deployed, you certainly do not want additional downtime for your business-critical systems, especially in the healthcare sector, industry, or other time-critical environments. Alternative to Lift and Shift as part of requirements definition However, sometimes Lift and Shift is the only solution, as this may involve political and commercial constraints, as well as time pressures. In any case, a thorough requirements definition should always be done for a data migration project before moving to a new environment. This involves reviewing the organization’s needs, including a business impact analysis , a target architecture and capacity plan , as well as a savings calculation and a continuously updated risk analysis , and considering a migration path for different components, which typically includes a number of preparatory performance optimizations, right-sizing, and workload consolidation at the server, instance, and database levels. This way, you will hit the target immediately with a very high percentage and avoid Lift and Shift problems. Summary FinOps develops a strong financial backbone for data migrations and keeps the budget under control, especially when moving to a public cloud like Azure. The reward is saving resources such as money, energy, and the environment. Sustainable, one piece at a time. Are you interested in SQL Governor, FinOps and data migrations? Please contact us! Jani K. Savolainen jani.savolainen@dbproservices.fi 0440353637 VP & Chairman DB Pro Services Oy
