Fisher were haunted by the return of one of their ex-players as they lost 2-1 to Beckenham at St Paul’s last Saturday – their fifth straight defeat.
Rob Brown gave the Fish a fourth-minute lead but Mathieu Ramsamy, who left the Rotherhithe club in the summer, equalised just four minutes later.
The visitors netted the winner ten minutes into the second half through Harvey Brand.
Allan Fenn’s side are seventeenth in the SCEFL Premier Division table.
Fisher hosts Chatham Town in the league this Saturday. Kick-off at St Paul’s is 3pm.