The use of the lake in the cloud has impacted an increase in cloud costs. I'm considering doing a TCO study to bring some workloads to our datacenter in house. Has anyone gone through or is going through this challenge?

Cloud FinOps is getting a lot of attention with increasing cloud costs. Costs can be optimized in multiple ways, with bringing the workloads in-house being one of the options. I recommend ensuring the study is extensive and identifying relevant workloads based on the value they provide back. This helped me iron out a lot of storage and compute costs that do not provide any ROI to the organization. When there is no value, there is no point in continuing to invest on them.

I also used products that are very effective in this and can share information if you like. 

Managing public cloud costs is a complex challenge, one that requires an expert approach. Since 2013, we’ve been deeply involved in migrating workloads for our customers (over 18.000 migrated), and the intricacies of tagging and controlling resources like VMs, storage, and load balancers have become increasingly apparent.

Our expertise in FinOps has highlighted several key areas where organizations often overlook cost management. For example, VM usage can easily escalate if not closely monitored. Similarly, storage solutions like S3 buckets often accumulate data without proper purge or size controls, leading to unnecessary expenses. Backups that aren't systematically removed and unchecked log growth are other frequent sources of cost inefficiency—logs, in particular, can be a significant issue as developers may not always manage their size effectively.

Our multicloud strategy has proven advantageous, as there are considerable cost variations among cloud providers. Leveraging these differences can offer significant savings. However, it’s important to avoid the common pitfall of attempting to replicate on-premises environments in the cloud, as it requires a different strategy to be cost-effective.

We have recently begun to migrate more data and workloads from our data center to the cloud. The cost is not what the cloud provider said it would be. I agree with the other comments that there needs to be constant monitoring of the cost. Also, I feel that you should bring in an expert on cloud optimization whether that is a FTE or you use a consulting company. 

Cost management in the cloud is a science and it needs to be done consistently to be effective.  Individuals for whom cloud computing is one of their responsibilities will likely not fully realize the benefits and opportunities that cloud computing creates.  It is sort of like going to visit another country vs. living in that country.  Tourists think they experienced the city or country by visiting for some days or weeks.  However, it is only the residents who truly understood.  Put it another way, spending a portion of ones time in cloud computing is like being a touris.

Certain layers of datalake may be more easily transferable for on premises or similar scenarios.  One example is to place your own datalake storage in a localzone data center for say AWS, then use this storage with everything else for the data lake sitting in AWS.   

It is a good exercise to go through as I agree many organizations do not calculate the costs of cloud correctly. There is also an understanding you must have on what can and cannot run in the cloud based upon your application profiles, etc. It's not as simple as picking up and moving as many think when you embark down this journey. In many cases the Cloud may cost you more, but may be offset in soft costs so that should be called out in any business case as well.