Yes, it’s extremely annoying. It should be optional in the first place. Let’s wait for the WP guys to fix this issue once and for all. I hope WP 4.9 will make it optional, or just completely remove it. This. Is. So. Annoying.!!!
For the meanwhile, you can use this plugin:
Or just wait for the future release of WordPress.
And nope. I am not alone. There are a lot of us:
And more. Oh please, Matt… for the sake of all the people of the universe, just disabled the visual editor in text widgets. T_T