Example: NaN Detection and Removal

Functions > Data Analysis > Outliers and NaN > Example: NaN Detection and Removal

Use the markNaN function to mark outliers as NaN (Not a Number) in data sets. Use the IsNaN, matchNaN and filterNaN functions to manage NaNs.

1. Read a file containing the number of sunspots recorded over the last three centuries, and plot the data.

<region id="ID0ERZMU" top="230.40000000000003" left="19.200000000000003" height="245" width="591.999999999999" xmlns="http://schemas.mathsoft.com/worksheet50">
  <plot background-type="white" origin-positioning="true">
    <xyPlot>
      <title class="- topic/title " wwtype:type="Paragraph" xmlns:wwtype="urn:WebWorks-Type-Schema" />
      <legend />
      <traces>
        <trace resultRef="76">
          <traceStyle color="#FFFF0000" symbol="x" line-weight="1" line-style="None">lines</traceStyle>
        </trace>
      </traces>
      <graph-size width="453.599999999999" height="187.4" />
      <axes>
        <xAxis rank="1" legend-position="PlotBoundaryBottom" start="1700" end="2000">
          <axisLine position="origin" positionticmark="0" legendWidth="100.183333333333" />
          <axisGrid>
            <gridFrequency>11</gridFrequency>
            <gridLabels display="true" />
            <gridLines />
            <tickMarks display="true" />
          </axisGrid>
          <axisLabel />
          <markers />
          <numberFormat>
            <decimal precision="3" show-trailing-zeros="false" radix="dec" imaginary-value="i" />
          </numberFormat>
          <plotEquations>
            <plotEquation>
              <math resultRef="81">
                <apply xmlns="http://schemas.mathsoft.com/math50">
                  <matcol />
                  <id xml:space="preserve">Data</id>
                  <real>0</real>
                </apply>
              </math>
              <math resultRef="82">
                <placeholder xmlns="http://schemas.mathsoft.com/math50" />
              </math>
            </plotEquation>
          </plotEquations>
          <xyDomain scale-type="linear" auto-scale="true">
            <startValue resultRef="78">
              <real xmlns="http://schemas.mathsoft.com/math50">1700</real>
            </startValue>
            <endValue resultRef="80">
              <real xmlns="http://schemas.mathsoft.com/math50">2000</real>
            </endValue>
          </xyDomain>
        </xAxis>
        <yAxis rank="1" legend-position="PlotBoundaryLeft" start="0" end="200">
          <axisLine position="left" positionticmark="-1" legendWidth="100.183333333333" />
          <axisGrid>
            <gridFrequency>11</gridFrequency>
            <gridLabels display="true" />
            <gridLines />
            <tickMarks display="true" />
          </axisGrid>
          <axisLabel />
          <markers />
          <numberFormat>
            <general precision="3" show-trailing-zeros="false" radix="dec" zero-threshold="15" imaginary-value="i" exponential-threshold="3" />
          </numberFormat>
          <plotEquations>
            <plotEquation>
              <math resultRef="87">
                <apply xmlns="http://schemas.mathsoft.com/math50">
                  <matcol />
                  <id xml:space="preserve">Data</id>
                  <real>1</real>
                </apply>
              </math>
              <math resultRef="88">
                <placeholder xmlns="http://schemas.mathsoft.com/math50" />
              </math>
            </plotEquation>
          </plotEquations>
          <xyDomain scale-type="linear" auto-scale="true">
            <startValue resultRef="84">
              <real xmlns="http://schemas.mathsoft.com/math50">0</real>
            </startValue>
            <endValue resultRef="86">
              <real xmlns="http://schemas.mathsoft.com/math50">200</real>
            </endValue>
          </xyDomain>
        </yAxis>
      </axes>
    </xyPlot>
  </plot>
</region>

2. Use the ThreeSigma function to find the indices of the outliers.

<region id="ID0ET2MU" top="576.00000000000011" left="38.400000000000006" height="46.376000000000005" width="268.09000000000003" xmlns="http://schemas.mathsoft.com/worksheet50">
  <math resultRef="97">
    <define xmlns="http://schemas.mathsoft.com/math50">
      <matrix rows="2" cols="1">
        <id labels="VARIABLE" xml:space="preserve">i</id>
        <id labels="VARIABLE" xml:space="preserve">j</id>
      </matrix>
      <eval>
        <apply>
          <matcol />
          <apply>
            <id labels="FUNCTION" xml:space="preserve" label-is-contextual="true">ThreeSigma</id>
            <apply>
              <matcol />
              <id labels="VARIABLE" xml:space="preserve" label-is-contextual="true">Data</id>
              <real>1</real>
            </apply>
          </apply>
          <real>0</real>
        </apply>
        <unitOverride>
          <placeholder />
        </unitOverride>
      </eval>
    </define>
    <resultFormat>
      <decimal precision="round-trip-decimal-places" show-trailing-zeros="false" radix="dec" imaginary-value="i" />
      <matrix size="12,12" offset="0,0" show-indices="false" expand-nested-arrays="false" />
    </resultFormat>
  </math>
</region>

The outliers are the number of sunspots recorded during the following years:

3. Use the markNaN function to mark as NaN the outliers in column 1 of the data.

Replacing data with NaNs indicates that a measurement was made, but the rows containing NaNs can be filtered out before processing.

4. Use the matchNaN function to find the indices of the spots that have been marked as NaN.

The data in rows 257 and 278 has been replaced by the built-in constant NaN:

5. Use the IsNaN function to check if the year 1957 has been marked as NaN in the Data and in the MarkedData sets.

<region id="ID0EVDNU" top="1267.2000000000003" left="38.400000000000006" height="38.952000000000005" width="173.48666666666665" xmlns="http://schemas.mathsoft.com/worksheet50">
  <math resultRef="109">
    <eval xmlns="http://schemas.mathsoft.com/math50">
      <apply>
        <id labels="FUNCTION" xml:space="preserve" label-is-contextual="true">IsNaN</id>
        <apply>
          <indexer />
          <parens>
            <apply>
              <matcol />
              <id labels="VARIABLE" xml:space="preserve" label-is-contextual="true">Data</id>
              <real>1</real>
            </apply>
          </parens>
          <real>257</real>
        </apply>
      </apply>
      <unitOverride>
        <placeholder />
      </unitOverride>
    </eval>
    <resultFormat>
      <decimal precision="round-trip-decimal-places" show-trailing-zeros="false" radix="dec" imaginary-value="i" />
      <matrix size="12,12" offset="0,0" show-indices="false" expand-nested-arrays="false" />
    </resultFormat>
  </math>
</region>

<region id="ID0EDENU" top="1305.6000000000001" left="38.400000000000006" height="38.952000000000005" width="227.67" xmlns="http://schemas.mathsoft.com/worksheet50">
  <math resultRef="111">
    <eval xmlns="http://schemas.mathsoft.com/math50">
      <apply>
        <id labels="FUNCTION" xml:space="preserve" label-is-contextual="true">IsNaN</id>
        <apply>
          <indexer />
          <parens>
            <apply>
              <matcol />
              <id labels="VARIABLE" xml:space="preserve" label-is-contextual="true">MarkedData</id>
              <real>1</real>
            </apply>
          </parens>
          <real>257</real>
        </apply>
      </apply>
      <unitOverride>
        <placeholder />
      </unitOverride>
    </eval>
    <resultFormat>
      <decimal precision="round-trip-decimal-places" show-trailing-zeros="false" radix="dec" imaginary-value="i" />
      <matrix size="12,12" offset="0,0" show-indices="false" expand-nested-arrays="false" />
    </resultFormat>
  </math>
</region>

6. Plot the new data set, and compare it with the old set.

<region id="ID0ETFNU" top="1401.6000000000001" left="19.200000000000003" height="264.2" width="595.599999999999" xmlns="http://schemas.mathsoft.com/worksheet50">
  <plot background-type="white" origin-positioning="true">
    <xyPlot>
      <title class="- topic/title " wwtype:type="Paragraph" xmlns:wwtype="urn:WebWorks-Type-Schema" />
      <legend />
      <traces>
        <trace resultRef="113">
          <traceStyle color="#FFFF0000" symbol="star" line-weight="1" line-style="None">lines</traceStyle>
        </trace>
        <trace resultRef="114">
          <traceStyle color="#FF0000FF" symbol="filledRhombus" line-weight="1" line-style="None">lines</traceStyle>
        </trace>
      </traces>
      <graph-size width="399.599999999999" height="187.4" />
      <axes>
        <xAxis rank="1" legend-position="PlotBoundaryBottom" start="1950" end="1970">
          <axisLine position="origin" positionticmark="0" legendWidth="100.183333333333" />
          <axisGrid>
            <gridFrequency>11</gridFrequency>
            <gridLabels display="true" />
            <gridLines />
            <tickMarks display="true" />
          </axisGrid>
          <axisLabel />
          <markers />
          <numberFormat>
            <decimal precision="3" show-trailing-zeros="false" radix="dec" imaginary-value="i" />
          </numberFormat>
          <plotEquations>
            <plotEquation>
              <math resultRef="119">
                <apply xmlns="http://schemas.mathsoft.com/math50">
                  <matcol />
                  <id xml:space="preserve">Data</id>
                  <real>0</real>
                </apply>
              </math>
              <math resultRef="120">
                <placeholder xmlns="http://schemas.mathsoft.com/math50" />
              </math>
            </plotEquation>
          </plotEquations>
          <xyDomain scale-type="linear" auto-scale="true">
            <startValue resultRef="116">
              <real xmlns="http://schemas.mathsoft.com/math50">1950</real>
            </startValue>
            <endValue resultRef="118">
              <real xmlns="http://schemas.mathsoft.com/math50">1970</real>
            </endValue>
          </xyDomain>
        </xAxis>
        <yAxis rank="1" legend-position="PlotBoundaryLeft" start="0" end="200">
          <axisLine position="origin" positionticmark="-1" legendWidth="154.293333333333" />
          <axisGrid>
            <gridFrequency>11</gridFrequency>
            <gridLabels display="true" />
            <gridLines />
            <tickMarks display="true" />
          </axisGrid>
          <axisLabel />
          <markers />
          <numberFormat>
            <general precision="3" show-trailing-zeros="false" radix="dec" zero-threshold="15" imaginary-value="i" exponential-threshold="3" />
          </numberFormat>
          <plotEquations>
            <plotEquation>
              <math resultRef="125">
                <apply xmlns="http://schemas.mathsoft.com/math50">
                  <matcol />
                  <id xml:space="preserve">Data</id>
                  <real>1</real>
                </apply>
              </math>
              <math resultRef="126">
                <placeholder xmlns="http://schemas.mathsoft.com/math50" />
              </math>
            </plotEquation>
            <plotEquation>
              <math resultRef="127">
                <apply xmlns="http://schemas.mathsoft.com/math50">
                  <matcol />
                  <id xml:space="preserve">MarkedData</id>
                  <real>1</real>
                </apply>
              </math>
              <math resultRef="128">
                <placeholder xmlns="http://schemas.mathsoft.com/math50" />
              </math>
            </plotEquation>
          </plotEquations>
          <xyDomain scale-type="linear" auto-scale="true">
            <startValue resultRef="122">
              <real xmlns="http://schemas.mathsoft.com/math50">0</real>
            </startValue>
            <endValue resultRef="124">
              <real xmlns="http://schemas.mathsoft.com/math50">200</real>
            </endValue>
          </xyDomain>
        </yAxis>
      </axes>
    </xyPlot>
  </plot>
</region>

The outliers from the original data set are not highlighted in blue, since the plot skips the NaN recorded in the MarkedData set.

7. Use the filterNaN function to filter the matrix MarkedData set to remove the rows containing NaNs.

8. Use the rows function to calculate the number of rows in the Data and FilteredData sets.

The number of rows in FilteredData has decreased by two.

9. Use the mean function to calculate the mean of the MarkedData and the FilteredData sets.

Statistics can be collected for the FilteredData set, but not for the MarkedData set.