For me this is y top 5 toughest.
1, Merion - toughest greens, top course management required
2. Olympic - hard greens and some hard fairways to hit
3. Oakmont- tough greens
4. RSG - winds can cause havoc
5. Congressional - tight fairways
Basically I recommend you play all courses, some shots you use on one course do not necessarily work on others so take notes of what works and where, especially the short game. All too often you find a player who is brilliant on one course and looks and plays like a hack on every other course.
This game is not easy and you are always learning so most importantly, enjoy the journey.
Most people find St Andrews front nine the easiest with Kiawah coming in second. However to lower averages sometimes It easier to do belter on the shorter nines (par 35 or less)