You have to have seen the map that is located in the locked drawer at bottom of the bed in Drake's room.
From the walkthrough:
Drake's room:
Use the lantern on the picture on the wall right of the desk. Click on the tower above the cliff on the painting. Press the button and a drawer below the bed is opened.
A picture of the lighthouse with a purple 1, green 2, yellow 3 and blue 4 above a metal canister is seen. Another picture of a series of walkways going towards under the lighthouse is depicted. Note that the start of the walkway is behind the lighthouse and goes to a cave under the lighthouse.
If you have seen the ladder down but can't go down that ladder - move the cursor and see if you can get the down arrow.