But the club are on the up under Smith, whose managerial career has also featured spells at Crawley Town, Corinthian Casuals, Dulwich Hamlet and Kingstonian.
In a statement, Walton and Hersham FC said the proposals would "force Swans to move into an ill-considered and highly problematic sharing arrangement with Walton Casuals FC and Walton Athletic Club at a very hard-to-get-to out of town site at Waterside Drive".