Small values resolving to 0 in results page.

I've been seeing an issue that occurs for some procedures. When there are very small values evaluated, like meter zeros or calculated hystereses, the values get reported on the results page (and, subsequently, our calibration reports) as 0. It is recorded, however, in the full results table (in the database) correctly.

I've attached a couple of images that should more clearly explain the problem.

Also, some procedures have really ugly formatting, while others line up nicely in the results page.

Is there any way to force a displayed resolution on the results page?

1 comment

Date Votes
Michael Johnston

The order of precedence for resolution that MET/CAL looks for what to use is as follows:

  1. Instrument FSC with a value in the Nominal Field (Evaluation Test Type)
  2. MEMC/MEMCX FSC with a value in the Nominal Field
  3. ACC FSC with a vaue in the Nominal Field
  4. SETUP FSC with a value in the Nominal Field
  5. MEMC/MEMCX Tolerance Field which includes a U (units from the nominal field) component

Are you using MEM or MEM1 to fill your nominal field for your evaluations? If you are and you don't have any of the other options filled, MET/CAL tries to "infer" what it thinks it should be. Using TSET UUT_RES = xxxxx should override any of the 5 options I listed there, but the one thing to remember that might be an issue is that UUT_RES is expressed in base units, not reported units. So to get a resolution of 0.001 mV, you would need to use UUT_RES = 0.000001.

I guess I'd need to see your code to get a better idea of what might be going wrong, but see if any of this helps.


Please sign in to leave a comment.