a few years ago I went there and stayed a few nights in my girlfriend's (now wife) apartment. Her old home turned into a guesthouse. There are lots of embassies on the island (German, Japanese) due to the opium war loss. The Japanese embassy is now an organ museum well worth a visit. If you walk a bit more just around the Japanese embassy, there is an artist who use the little bones from fish and make it into very nice art work picture. You can see the man himself and his pretty assistants. He was a very nice guy to talk to and you really will see something that you won't see elsewhere. Pay a very small fee to have a look.
In the back of the island there are people living there. There are many places to eat.
In the market there is a man who make maltose lollipop for you, with picture pattern of your choice. Buy one for your girlfriend.
Also there is a piano museum. The whole island is called the "Piano Island" because of a music school there and the little place has produced many famous musicians over the years.
Unfortunately the place is spoiled by far too many tourists. But it is still possible to be alone and do your own thing.