Jatinder Singh, from Smethwick, was summoned to serve as a juror at Birmingham Crown Court on Monday
But, he said, a security guard refused him entry at the court over his kirpan, the sword carried by all Sikhs.
The Ministry of Justice (MoJ) said Mr Singh was released from his duties as there was a surplus of required jurors.
Meanwhile, His Majesty's Courts and Tribunals Service (HMCTS) has apologised to Mr Singh.
Khalsa Sikhs carry the five Ks with them at all times, as a symbol of their faith.
These include Kesh or uncut hair; Kara which is a a steel bracelet; the Kanga, a wooden comb; Kacca or cotton underwear and the Kirpan
Mr Singh, who has served as president at Guru Nanak Gurdwara in Smethwick and as secretary general of the Sikh Council UK, said this was the second time he has been summoned for jury service, the first passing with no issue.
On this occasion, he entered the morning session with no problems, but when he tried to return to the court after lunch was pulled aside by a security guard and told he could not go in.
"The security guard said I could take [my kirpan] off and leave it with him and collect it at the end of the day," he said.
"I felt like a child who has gone to school and taken something they shouldn't and had it confiscated.
"To have that happen to me, I felt embarrassed, I felt discriminated against, I didn't expect it to happen to me."
He called for the (MoJ) to work with Sikh and other religious organisations to create easily accessible guidelines that can be provided to staff.
Dabinderjit Singh, the principal adviser to the Sikh Federation UK said it had written to Justice Minister Alex Chalk asking him to condemn the treatment of Mr Singh.
The MoJ said members of the Sikh community wishing to enter a court building could bring in a Kirpan which was not more than six inches long (15cm) and with a blade no more than five inches (12cm) in length - which Mr Singh said his was.
A spokesperson for HMCTS added: "We have apologised to Mr Singh for any distress caused and have reminded our contracted security officers of the correct steps to take to prevent this incident from happening again."