The velvet room isn't that hard to get into. Flashback to Persona 1 or 2 where it was just a physical location you could move to.
Otherwise, there are clearly a lot of exceptions to the supposed rule. Marie was let into the Velvet room almost seemingly on a whim, Teddie as well wandered into there, and can't forget about PQ where the entire cast of P3 just managed to enter somehow. The same thing happened with the entire cast of P5. The rule seems more like a guideline then an actual rule.