Whiskey Point is a quite surfable and moderately swimmable point about 12 km from Arugam Bay. It's a bit of a trek, but worth it for some great waves and the chill cafe/hotel SaBaBa's.
For surfers, Whiskey Point offers beginner to intermediate waves. They also break somewhat perpendicular to the beach. This means that you can catch waves without swimming out too far. You can just walk along the beach and catch them where they break, at the point. The waves are also gentle enough that surfers quite happily bob and let them pass until a good one arrives.
The waves, however, are not gentle to swimmers. There is nothing to be afraid of here, it's a bay without rocks or undertow, but the waves are powerful, especially if you're afraid of them. If you're comfortable bodyboarding or just getting thrown around it can be quite fun, but also exhausting. This is not a place for a casual dip in the ocean.
Besides swimming, however, the rocks on the point make for an awesome if not insanely comfortable sofa. You can climb to the top and watch the surfers, something recommended when it's not so blazingly hot, either around sunrise or sunset. The sunrises from here are amazing (as the sun rises in the east, over the ocean) and the sunsets aren't so bad either.
In terms of amenities, SaBaBa's cafe is nearby. Run by the younger brother of Mambo (who runs a similar joint on Main Point, as well as being a captain of industry in Hikkaduwa), SaBaBa's has a simple but great design, lending itself to chilling and watching the waves. They also have shaded daybeds on the beach, perfect for reading. They have cold beer and a decent selection of juices and bites.
The waves are not dangerous (no undertow or rocks), but they are strong. You can safely clamber up the rocks if you just want to watch.