A custom was a long-standing way of life or belief of a species. Beginning as a new practice or usage, a custom could, with the passage of many generations, become a tradition. As well, a custom could become a force of law. Custom was a part of culture. (TOS: "The Cage"; TNG: "Half a Life")
The records of the USS Enterprise revealed to the Talosians that the customs and history of Humans showed a unique hatred of captivity. Even when the captivity was pleasant and benevolent, Humans preferred death. (TOS: "The Cage")
A Human custom was "to try and avoid misunderstanding whenever possible." (TOS: "The Corbomite Maneuver")
On Amerind, it was against custom to interrupt a tribal elder at council, even for a medicine chief. (TOS: "The Paradise Syndrome")
In 2364, Ligonian custom made it clear that a visiting leader became an honored guest. (TNG: "Code of Honor")