Precision, Performance, Confidence. ™

Measurement Resolution in MET/CAL Software

Controlling resolution for results generated by instrument evaluation

The Device Under Test (DUT) resolution is determined by the following rules:

  • If the instrument FSC Nominal field contains a value, use it to determine the resolution.

Example

5700    400    350.0mV          2.9U          60H                   2W

 will give one significant digit in the results.

IEEE           [T30000]OUT 2Mv, 10Hz; OPER;*OPC?[I]

5790     2.2    2.00000mV     0.305%     10H        I1        2W

  • If the instrument FSC Nominal field does not contain a value format control is not possible. An instrument evaluation test without a Nominal field value must be broken into an instrument setup and a MEMC or MEMCX FSC to properly control the resolution (See MEMC below)

Controlling Resolution for Results Generated by the MEMC/MEMCX FSC

The resolution is determined by the following rules:

  • If the MEMC or MEMCX Nominal field contains a value, use it to determine the resolution. This case is not typically used. If the MEMC Nominal field contains a value, it is taken as the DUT Indicated value. A constant DUT Indicated value only makes sense for evaluation of a source DUT.  Source DUTs are typically tested using instrument evaluations such as the 5790 statement in the example above. However, this case must be used if a User-configured Device (standard for which there is no FSC) is used to evaluate a source DUT.

Example

IEEE          AP-12DB, SP84

 IEEE         [@436][I]

 MEMC     -12.0D        1.0U

  • Otherwise, if there is a prior ACC statement, use the ACC Nominal field value to determine the resolution. This is the case when the test units are not the same as the units of the standard.

Example

5700         2V                                                           60H                      S  2W

 

DISP        Press the MIN MAX button to display minimum values

ACC         -2.828Vp 75P% 7e-6U

MEMI        Enter DUT reading in volts peak

MEMCX     Vp                0.045U

  • Otherwise, if there was a prior setup statement, but no prior ACC statement, use the Nominal field value of the setup statement to determine the resolution. This is the typical case for closed-loop meter procedures or non-closed loop procedures when the Nominal field of the instrument FSC does not contain a value. The value is taken from memory register MEM.

Example

5700              1.00000V                  200H           S     2W

IEEE               ?[I]

MEMCX      2     v           0.00150U  200H

In this case the resolution is 5 significant digits

MATH           MEM = 350

5700             mV              60H               S 2W

MEMI            Enter the DUT reading in microfarads

MEMCX          1        uF           1%

In this case the resolution is no significant digits

  • Otherwise, if the MEMC Tolerance is specified in units of the Nominal “U”, use it to determine the resolution. This is the case when a test is performed with no input connected to the DUT.

Example

DISP          Disconnect the 5700A from the DUT.

DISP           Press uF

MATH         MEM1 = 0.0

MEMI          Enter DUT reading in microfarads

MEMCX  1                 uF              0.001U

  • Finally, if none of the above conditions are met, MET/CAL will format the results with up to 10 digits to the right of the decimal point to preserve non-zero digits. However, trailing zeros to the right of the last non-zero digit will be stripped off.
Have more questions? Submit a request

Comments

Please sign in to leave a comment.