Actually.. the answer is No.. you can turn off the music and still have sound effects such as, hitting the ball, the ball hitting the ground, the ball going in the cup, the ball hitting the flag, sound effects for trophies, and the "its your turn (or you have 30 seconds left) ding alert.
However if you turn the Music off you lose sounds effects such as, the slight ambient wind (open air) effect, and the heartbeat effect in blitz and CTTH games.
You may lose other sounds but I'm not sure if there are any others since I usually have the music turned off so I dont remember all the sounds with it on.
The reason I say No is because the OP specifically stated "ambient" sounds.. which the open air (wind) whatever you wanna call it sound and the heartbeat are ambient effects.. while the other mentions are action related effects.