Sources say that the baby arrived without much fanfare, with Gayford driving Ardern to a public hospital.
Ardern found out she was pregnant on October 13, just six days before she was propelled into the job when Peters, the leader of the New Zealand First Party, announced he was siding with her Labour Party in post-election negotiations. "Feeling very lucky to have a healthy baby girl".
Mr Gayford will assume the role of full-time carer of the baby when Ms Ardern returns to prime ministerial duties.
"It's wonderful", she said.
Ardern is normally based in Wellington but relocated to be closer to Auckland in preparation for the birth.
Spanish MP Carolina Bescansa breastfed in her national legislature in early 2016, while the issue of juggling children and a political career hit the headlines in Australia previous year when senator Larissa Waters became the first woman in the country's history to breastfeed in the chamber.
Former prime minister Helen Clark praised Ardern and her partner's parenting arrangement as an example of gender equality after the birth.
The baby's name is still unknown - the couple recently admitted they were struggling with choosing a name.More news: Trump's migrant policy: First blowback, then about-face
More news: Spain Face Buoyant Iran
More news: Under fire Eddie Jones responds to public backing from RFU
Recently describing the impending handing over of the reins as a "snooze fest", leader Simon Bridges will of course still be keeping a watching brief on Winston Peters as prime minister, ready to pounce on any mis-step.
"Natalie and I are delighted for Jacinda and Clarke".
"Being parents is a joy and a privilege and we are sure your new arrival will fill your lives with all the happiness our children have ours", Bridges said.
The US ambassador to New Zealand, Scott Brown congratulated Ardern both in English and in Maori on Thursday.
Ardern's pregnancy came as a surprise to her and Gayford, but they took it in their stride.
Jacinda Ardern continued to work as Prime Minister right up until the birth.
Her office confirmed the Prime Minister was in labour on Thursday morning, reportedly already four days past her due date.
Ardern's Australian counterpart Malcolm Turnbull told ABC Radio he was "really thrilled", while New Zealand opposition leader Simon Bridges offered congratulations.