Police responding to a 911 call about an assault in progress just after 9.15pm arrived to find 53-year-old Cali with six gunshot wounds to the torso.
Police confirmed the identity of Cali, who was pronounced dead at Staten Island University Hospital.
No arrests have been made.
The Gambino crime boss was assassinated outside Sparks Steakhouse in Manhattan in 1985.
Since Gotti's downfall, the Gambinos - and other mafia families - have lost much of the power they once had over politicians, judges and labor unions.
Cali himself was jailed for 16 months for his connection to an extortion scheme surrounding an attempt to build a NASCAR track on Staten Island.More news: Oil prices rise on OPEC's supply cuts and healthy demand
More news: Fortnite Now Forces Cross-Play Between PS4 And Xbox One
More news: Lawyer told Cohen he could 'sleep well' after talking to Giuliani: emails
In 2012, Cali was identified as the new underboss of the Gambino crime family.
NY media say it is the first targeted killing of a mob boss in the city since 1985.
After reportedly taking the reins in 2015, Cali was described by mafia watchers as the opposite of Gotti because "no-one ever sees him", the New York Post reported.
According to a 2008 Advance report, Cali - who also went by "Franky Boy", - was born in Brooklyn and married into the Inzerillo family of Palermo and cultivated close ties with members of the Siderno cartel in Italy.
The Gambino family was once one of the most notorious crime syndicates in America and one of New York's five major crime families.
Two years later, reputed Gambino soldier Frank Fappiano of Tottenville was nabbed by the feds, then gave up Cali's name as a "wiseguy", an Italian media outlet reported at the time.
The New York Daily News reported in 2008 that Cali was among 62 people arrested on federal racketeering charges and he later pleaded guilty to conspiring to extort money.