If it can't talk to anything, ntpd just sits quietly in the background and does nothing bad. It isn't like it says, "oh, I can't find a server, so I'll just randomly alter the system time, that will teach him!"