Don’t say: 
– I’m too scary to go bungee jumping.
– He was scary of the loud noise.

Say this:
– I’m too scared to go bungee jumping.
– He was scared of the loud noise.

Remember:

• Scared (used as an adjective) is a feeling – similar to afraid.  I feel scared.  She is scared of bees.
• Scary is the reason for this feeling.  Going bungee jumping is scary.  Sharks are scary.