Do one thing, when you take take taxi, ask him about timing of buses for Ghazipur, he will tell you or if you don't have much luggage then get down at Roadways bus station which is just infront of Varanasi Junction and check in enquiry. I have taken bus for Ghazipur once around 0600 from Roadways Bus Stand.