0

Limiting resolution based on uncertainty magnitude

This keeps coming up, and I thought I'd share this here.

A handy way of limiting reported resolution for a RSLT line, based on the magnitude of an uncertainty.

 


# Limit resolution from a string using corresponding uncertainty
1.056 MATH string = "1.0053613,50.03521810262,100.0512026621,10544432244"
1.057 MATH uncert = "0.0034, 0.00045, 0.12, 44432244"

1.058 MATH field = 1
1.059 DO

1.060 MATH field_val = fld(string, field, ",")
1.061 MATH uncert_field = fld(uncert, field, ",")
1.062 MATH intlog = INT(LOG(fld(uncert, field, ","))) - 1
1.063 MATH value_to_report = crnd(field_val, 10^(intlog))
1.064 RSLT =[v field_val], [v uncert_field], [v value_to_report]

1.065 MATH field = field + 1
1.066 UNTIL field > findi(string, ",", 0) +1

Please sign in to leave a comment.