Startups: Here are your hosting options

Here are your hosting options:-

Dedicated server in colo facility: Highest up-front cost, you need to be operations person too, Lowest latency, best performance per $, entire machine is yours so blazingly fast when lightly loaded, can run virtual servers, cannot scale up or down rapidly based on demand. Typically given 5+ IP addresses so you have lots of flexibility in hosting multiple sites on the server.

Dedicated server rented: No up-front cost, higher cost per month than dedicated, easier to upgrade to new hardware.  Someone else's problem when it breaks. Typically come with a single IP address.

Virtual Servers: Can add new server instances / environments easily.  You can run these on your own server or rent them from someone else.  Great for load balancing to get the most out of your hardware. Typically somewhat limited in how much RAM you get.

Cloud Servers: Just like having a virtual server but you can scale up or down easily to add new instances.  Highest cost per CPU cycle.  You get what you pay for but no more - even if the server you are on has spare CPU cycles you are typically limited to the fraction you are paying for.  Shared network, virtual disk, ... means it will never be as fast as a dedicated server.

In the end after trying cloud servers, virtual servers and dedicated servers I went with the first option: dedicated high performance servers with SSD drives in them hosted in a colo-facility. This setup outperform all the other options significantly providing lower latency (better response times) and the ability to run multiple virtual server instances on them for complete isolation between services, and in the long run it will be cheaper also. Later when we need to scale up or down based on time of day we will probably combine this approach with some extra cloud servers but at the quietest times of day I would expect to have no cloud instances running.