In a PN junction : -

When the junction forms, holes diffuse from the p-side to the n-side and electrons from the n-side to the p-side.
This leaves positively charged immobile donor ions on the n-side and negatively charged acceptor ions on the p-side of the junction (the depletion layer).
The positive charge on the n-side and negative charge on the p-side create an electric field from n to p. So the n-side is at a higher potential than the p-side; this is the barrier potential.

So there is high potential at the N side and low potential at the P side.