modified code:

 

int a = 3;
int b;
int h = 5;

 

...

 

  Serial.print("h = ");
  Serial.println(h);
 
  b = sqrt(h*h - a*a);
 
  Serial.print("b = ");
  Serial.println(b);

 

This program ends up with the serial monitor reading:

 

Lets calculate a leg
a = 3
h = 5
b = 4