Why are non-IT executives still pursuing shadow IT in spite of consistent evidence that what often starts well, usually gets departments and often companies into trouble later? Why not just ask for help early on?

I think to answer you have to define what IT is. In the past business and IT were relatively siloed. Over time the line blurred as things such as ERP became more prevalent.

Today IT and operations lines are blurred, and, in many cases modernizing technology can make more agile business less dependent on central IT. We have to assume that the resources available in IT are actually capable of executing the job that needs to be solved today (or tomorrow) before we say that shadow IT is a mistake.

That is not to say that shadow IT is the answer, but in many cases org structures are sticky. There are still lots of organizations where there is someone managing on-prem infrastructure because the cloud didn’t exist when it was put into place (and you needed someone to actually be there). In this example, starting from scratch you would probably go to the cloud—to do that assumes the skillset of managing an on-premise environment, where talent is geographically constrained and the tasks have more of a manual component, is the same as managing cloud infrastructure, which is more about strategy (since many of the procurement pieces are taken care of by the vendor).

It can very well be that the business is not thinking forward and being tactical when they create shadow IT. At the same time it could very well be that the business doesn’t think IT is capable because it isn’t. As more roles require a cross-disciplinary skillset this problem is only likely to become worse.

In my experience, shadow IT results from misaligned incentives between organizations - the business unit has a KPI to meet short term. IT doesn't have the resources (budget, capacity, technology, vendors, or others) to help the business unit achieve results.

Some thoughts to help mitigate these issues:
- Understand and adopt business KPIs in the IT organization
- Anticipate and plan demand based on future business plans
- Ensure you have slack to respond to change
- Streamline your project onboarding process

And if all else fails, make sure you engage early and build resources to absorb the need quickly.

The underlying causes are the same as everyone else noted here - 1- Central IT department lacks flexibility 2-Process to onboard a new solution is too cumbersome 3-IT lacks the technical depth for the solution a specific LoB needs 4-LoB is too new and may not know the process to even ask for help 5-Business pressure to meet deadlines. The theme is the - there is a clear disconnect or friction, and they have a need that must be met. Perhaps they should look for a start elsewhere. I am sure everything goes through CFO office. Why not engage them with a clear demand of "I need X to meet Y in Z timeframe"? This will allow IT to come to the table with options and be accountable for business needs. 

Shadow IT is a big security risk to the organization. CIO/CISO must have good controls to detect, identify and manage shadow IT.

Its important to know why and how shadow IT gets formed. IT office cant not dictate. Rather IT should create a solid framework to regularise such demands in a more secured, fast and cost effective manner. I am sure gaps will always exists which needs to be fixed, 
but IT cant disown shadow IT.