I don't remember exactly but I believe it is stuck on or in one of the building nearby where the fisherman is.
Some extra info: hit all the barrels in the Fisherman Town and Waterfall Of the Heavens to get them red side up. This will make the golden fishhook appear.