Please enable JavaScript to view this site.

Knowledge Base

Navigation: Scripting Language > Functions

precision

We've made some changes. Use 'Ctrl-F5' to clear your browser cache.
Scroll Prev Top Next More

Truncates a number to the precision specified by the user. The precision is specified by the desired number of significant figures used when rounding the number.

 

Supported Product: INTERCONNECT

 

Syntax

Description

out = precision (y,p);

Truncates y to a user defined precision p. Where p is the number of significant figures.

 

Example

Show different values of ‘pi’ depending on the user defined precision.

?pi;

result: 

3.14159  

 

?precision(pi,2);

result: 

3.1 

 

?precision(pi,3);

result: 

3.14 

 

?precision(pi,4);

result: 

3.142

 

?precision(pi/100,4);

result: 

0.03142 

 

See Also

Functions, round

Copyright Lumerical Inc. | Privacy | Site Map