Showing posts with label CSV. Show all posts
Showing posts with label CSV. Show all posts

Tuesday, 7 July 2026

Spot The Bug: It's a Bug! It's a Plane!

Well hello, readers! Time for some more Spot The Bug!

Go away or be
squashed, bugs!


This episode revolved around NodeJS. My code was supposed to read a CSV file and display the contents in table format. The most basic shit ever. Somehow I managed to screw it up!

This was the script that would be executed. Basically, I would load the file tbl.csv, from the file system, and parse it as CSV text. And make sure each row got pushed into an array. Then after that, a HTML table would be created from the CSV rows, and inserted into the view.

app.js
var express = require("express");

var app = express();

var handlebars = require("express-handlebars").create({defaultLayout:"main"});
app.engine("handlebars", handlebars.engine);

app.set("view engine", "handlebars");
app.set("port", process.env.PORT || 3000);

app.use(express.json());
app.use(express.urlencoded({ extended: true }));

app.use(express.static("assets"));

const fs = require("fs");
const csv = require("csv-parser");

let csvContent = [];

function loadFileCSV(filePath) {
   return new Promise((resolve, reject) => {
      const resultsCSV = [];

      fs.createReadStream(filePath)
      .pipe(csv())
      .on("data", (row) => {
         resultsCSV.push(row);
      })
      .on("end", () => resolve(resultsCSV))
      .on("error", (err) => reject(err));
   });
}

async function loadTableCSV() {
   try {
     csvContent = await loadFileCSV("assets/csv/tbl.csv");
   } catch (err) {
     throw new Error("Error reading CSV.");
     console.error("Error reading CSV:", err);
   }
}  

app.get("/superman", (req, res)=> {
  loadTableCSV();
  
  let csvTable = "<table><tr><td>Title</t><td>Year</t><td>Actor</t></tr>";

  for (let i = 0; i < csvContent.length; i++)
  {
    csvTable += "<tr><td>" + csvContent[i].Title + "</t><td>" + csvContent[i].Year + "</t><td>" + csvContent[i].Actor + "</t></tr>";
  }  

  csvTable += "</table>";

  res.render("superman", { table: csvTable });
});

app.use((req, res, next)=> {
  res.status(404);
  res.render("404");
});

app.use((err, req, res, next)=> {
  res.status(500);
  res.render("500", { errorMessage: err.code });
});

app.listen(app.get("port"), ()=> {

});


This was the view.

views/layouts/main.handlebars
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Superman</title>

    <link rel="stylesheet" type="text/css" href="css/styles.css">
  </head>
  <body>
    <h1>SUPERMAN DATA</h1>
    <div class="content">
      {{{ body }}}  
    </div>    
  </body>
</html>


views/superman.handlebars
<div>
{{{ table }}}
</div>

And this was the data.

assets/csv/tbl.csv
Title,Year,Actor
Superman/Atom Man vs. Superman,1948-1950,Kirk Alyn
Superman and the Mole Men/Adventures of Superman,1951-1958,George Reeves
The Adventures of Superboy (Pilot),1961,Johnny Rockwell
It's a Bird... It's a Plane... It's Superman,1975,David Wilson
Superman I-IV,1978-1987,Christopher Reeve
Superboy (TV Series),1988-1992,J.H. Newton/G. Christopher
Lois & Clark: The New Adventures of Superman,1993-1997,Dean Cain
Smallville,2001-2011,Tom Welling
Superman Returns,2006,Brandon Routh
Man of Steel/BvS/Justice League,2013-2017,Henry Cavill
Supergirl/Arrowverse/Superman & Lois,2016-2024,Tyler Hoechlin
Titans,2019-2023,Joshua Orpin
The Flash,2023,Nicolas Cage
Superman,2025,David Corenswet


What Went Wrong

So... wow. Totally nothing. No error, even. Why?



Why It Went Wrong

Notice that loadTableCSV() is an async function. Which means any call to loadTableCSV() better be async as well, or nothing will be returned. What happened was that I called loadTableCSV() and then went on to use csvContent without realizing that its value was as I initialized - an empty array. Because I didn't wait for the CSV data to load before using csvContent!
app.get("/superman", (req, res)=> {
  loadTableCSV();


Eventually, if I waited long enough, the data might show. But that would be scant consolation and something was still obviously wrong.

How I Fixed It

As it turned out, that was all that was needed. Change the callback signature to an async one, and the function call to loadTableCSV(), to an await function call.
app.get("/superman", async(req, res)=> {
  await loadTableCSV();


And the Superman table appeared!


Moral of the Story

Async is a tricky business. JavaScript, being async, is a tricky business by extension. And when you have NodeJS which is JavaScript in the front and back, it's a doubly tricky business.

Up up and away,
T___T

Sunday, 22 March 2026

Web Tutorial: The GitHub Commit Line Chart (Part 1/2)

Earlier this month, we took a look at my contributions over ten years of GitHub usage. I have the data, and we're going to do some data visualization on this one! Here's a sample of what hcdata_github.csv looks like. For the purpose of this exercise, I saved it at this link.
Year,Month,Contributions
2016,1,0
2016,2,4
2016,3,2
2016,4,2
2016,5,2
2016,6,7
2016,7,6
2016,8,6
2016,9,3
2016,10,0
2016,11,0
2016,12,7
2017,1,5
2017,2,3
2017,3,17
...


For this, we want something quick and dirty, so HighCharts it is! Here, we have some boilerplate HTML. Note the script link to the HighCharts library.
<!DOCTYPE html>
<html>
  <head>
    <title>GitHub Contributions</title>

    <style>
  
    </style>

    <script src="https://code.highcharts.com/highcharts.js"></script>

    <script>

    </script>
  </head>

  <body>

  </body>
</html>


We'll have two divs - ids container and dashboard respectively. I've included the styling. They both take up full screen width, though container has a bigger height than dashboard. dashboard has an additional specification to say text must be aligned in the middle (this will be relevant very soon). I've set divs to have a red outline, temporarily, so we can have a better visual.
<!DOCTYPE html>
<html>
  <head>
    <title>GitHub Contributions</title>

    <style>
      div { outline:1px solid rgb(255, 0, 0); }

      #container
      {
        width: 100%;
        height: 600px;
      }

      #dashboard
      {
        width: 100%;
        height: 100px;
        text-align: center;
      }
    </style>

    <script src="https://code.highcharts.com/highcharts.js"></script>

    <script>

    </script>
  </head>

  <body>
    <div id="container">

    </div>

    <div id="dashboard">
    
    </div>
  </body>
</html>


Simple enough so far?


Over here in the dashboard div, we add a checkbox, cbSeparateYears, within a label tag.
<div id="dashboard">
    <label for="rngYearFrom">
        <input type="checkbox" id="cbSeparateYears">
        SEPARATE YEARS
    </label>
    <br />  
</div>


And two sliders, rngYearFrom and rngYearTo, also within label tags. Beside each slider we have corresponding output tags.
<div id="dashboard">
    <label for="rngYearFrom">
        <input type="checkbox" id="cbSeparateYears">
        SEPARATE YEARS
    </label>
    <br />
    <label for="rngYearFrom">
        FROM
        <input id="rngYearFrom" type="range" />
        <output id="opYearFrom" for="rngYearFrom"></output>
    </label>
    <br />
    <label for="rngYearTo">
        TO
        <input id="rngYearTo" type="range" />
        <output id="opYearTo" for="rngYearTo"></output>
    </label>      
</div>


This styling here for labels is just meant to align stuff nicely at the bottom of the chart. Basically, the labels have a fixed width and are aligned right... within that fixed width of 20em. And then the whole 20em worth of labels is aligned smack in the middle of dashboard! Remember we set text-align to middle for dashboard? I also set the color to orange. Sorry, the color just speaks to me, y'know?
<style>
    div { outline:1px solid rgb(255, 0, 0); }

    #container
    {
        width: 100%;
        height: 600px;
    }

    #dashboard
    {
        width: 100%;
        height: 100px;
        text-align: center;
    }

    label
    {
        display: inline-block;
        width: 20em;
        font-family: verdana;
        color: rgba(200, 150, 0, 1);
        text-align: right;
    }
</style>


OK, so here's a preview. The output tags won't be visible simply because they have no values yet.


We want these sliders to behave a certain way. They will both call the function setYear(), but with different arguments.
<label for="rngYearFrom">
  FROM
  <input id="rngYearFrom" type="range" oninput="setYear('from')" />
  <output id="opYearFrom" for="rngYearFrom"></output>
</label>
<br />
<label for="rngYearTo">
  TO
  <input id="rngYearTo" type="range" oninput="setYear('to')" />
  <output id="opYearTo" for="rngYearTo"></output>
</label>  


Start writing some code to handle events when the page loads. At the same time, we'll define the setYear() function. It has a parameter, target. Also, declare currentData and yearSeriesData as empty arrays.
<script>
    document.addEventListener("DOMContentLoaded", function () {

      });      
    })
;

    function setYear(target)
    {

    }

    let currentData = [];
    let yearSeriesData = [];
</script>


When the page loads, we want to get the data, and populate the min and max attributes of rngDateFrom and rngDateTo, with the minimum and maximum value of the Year column. No point allowing the user to select an invalid year, amirite?! So first, we run fetch() to get the CSV data from the URL I've saved it in.
document.addEventListener("DOMContentLoaded", function () {
    fetch("http://www.teochewthunder.com/demo/hc_github/hcdata_github.csv")
  });      
});


We use then() to grab the output response once the data is loaded, and resolve it to text using the text() method.
document.addEventListener("DOMContentLoaded", function () {
    fetch("http://www.teochewthunder.com/demo/hc_github/hcdata_github.csv")
    .then(response => response.text())
  });      
});


Once that's done, the next step is to use a chained then() method call to grab the output, csvData, and start working on it.
document.addEventListener("DOMContentLoaded", function () {
  fetch("http://www.teochewthunder.com/demo/hc_github/hcdata_github.csv")
  .then(response => response.text())
  .then(csvData => {
    
  });

});


We first want to declare rows as an array obtained from running the split() method on csvData, splitting by newlines in the CSV text content.
document.addEventListener("DOMContentLoaded", function () {
  fetch("http://www.teochewthunder.com/demo/hc_github/hcdata_github.csv")
  .then(response => response.text())
  .then(csvData => {
    const rows = csvData.split("\n");  
  });
});


And now we can fill up currentData. Each element of rows is a CSV line, separated by commas. We first use the slice() method on rows, with the argument 1, to only take into account all the rows after the first one, which is the header. And then we run the map() method on the result to iterate through it.
document.addEventListener("DOMContentLoaded", function () {
  fetch("http://www.teochewthunder.com/demo/hc_github/hcdata_github.csv")
  .then(response => response.text())
  .then(csvData => {
    const rows = csvData.split("\n");  

    currentData = rows.slice(1).map(row => {

    });

  });
});


We define cols by running the split() method on row, with a comma as the argument. We declare year, month and contribution according to which part of cols we are referencing, and ensure that the result is an integer by using the parseInt() function. Then we return the array of all these. In short, we return cols, but with the values converted to integers.
document.addEventListener("DOMContentLoaded", function () {
  fetch("http://www.teochewthunder.com/demo/hc_github/hcdata_github.csv")
  .then(response => response.text())
  .then(csvData => {
    const rows = csvData.split("\n");  

    currentData = rows.slice(1).map(row => {
      const cols = row.split(",");
      var year = parseInt(cols[0]);
      var month = parseInt(cols[1]);
      var contributions = parseInt(cols[2]);

      return [year, month, contributions];

    });
  });
});


Now define years. This is an array that contains all the years in the dataset, currentData. For this, we run the map() method on currentData and just get the first column (index 0) as the value.
document.addEventListener("DOMContentLoaded", function () {
  fetch("http://www.teochewthunder.com/demo/hc_github/hcdata_github.csv")
  .then(response => response.text())
  .then(csvData => {
    const rows = csvData.split("\n");  

    currentData = rows.slice(1).map(row => {
      const cols = row.split(",");
      var year = parseInt(cols[0]);
      var month = parseInt(cols[1]);
      var contributions = parseInt(cols[2]);

      return [year, month, contributions];
    });

    var years = currentData.map(col => col[0]);
  });
});


Now that we have the array years, getting the minimum and maximum values is a simple matter of using the min() and max() methods of the Math object, and passing in as an argument all the values of years. Note the use of the Spread Syntax here.
document.addEventListener("DOMContentLoaded", function () {
  fetch("http://www.teochewthunder.com/demo/hc_github/hcdata_github.csv")
  .then(response => response.text())
  .then(csvData => {
    const rows = csvData.split("\n");  

    currentData = rows.slice(1).map(row => {
      const cols = row.split(",");
      var year = parseInt(cols[0]);
      var month = parseInt(cols[1]);
      var contributions = parseInt(cols[2]);

      return [year, month, contributions];
    });

    var years = currentData.map(col => col[0]);
    var yearMin = Math.min(...years);
    var yearMax = Math.max(...years);

  });
});


Here, we grab the DOM elements from the inputs and output.
document.addEventListener("DOMContentLoaded", function () {
  fetch("http://www.teochewthunder.com/demo/hc_github/hcdata_github.csv")
  .then(response => response.text())
  .then(csvData => {
    const rows = csvData.split("\n");  

    currentData = rows.slice(1).map(row => {
      const cols = row.split(",");
      var year = parseInt(cols[0]);
      var month = parseInt(cols[1]);
      var contributions = parseInt(cols[2]);

      return [year, month, contributions];
    });

    var years = currentData.map(col => col[0]);
    var yearMin = Math.min(...years);
    var yearMax = Math.max(...years);

    var rngYearFrom = document.getElementById("rngYearFrom");
    var rngYearTo = document.getElementById("rngYearTo");
    var opYearFrom = document.getElementById("opYearFrom");
    var opYearTo = document.getElementById("opYearTo");

  });
});


We then declare and set the range. yearFrom and yearTo are both set to yearMin. For the sliders, the min attributes of rngYearFrom and rngYearTo are set to yearMin, and the max attributes of rngYearFrom and rngYearTo are set to yearMax. The values of the sliders, as well as the outputs, are set to yearFrom and yearTo.
document.addEventListener("DOMContentLoaded", function () {
  fetch("http://www.teochewthunder.com/demo/hc_github/hcdata_github.csv")
  .then(response => response.text())
  .then(csvData => {
    const rows = csvData.split("\n");  

    currentData = rows.slice(1).map(row => {
      const cols = row.split(",");
      var year = parseInt(cols[0]);
      var month = parseInt(cols[1]);
      var contributions = parseInt(cols[2]);

      return [year, month, contributions];
    });

    var years = currentData.map(col => col[0]);
    var yearMin = Math.min(...years);
    var yearMax = Math.max(...years);

    var rngYearFrom = document.getElementById("rngYearFrom");
    var rngYearTo = document.getElementById("rngYearTo");
    var opYearFrom = document.getElementById("opYearFrom");
    var opYearTo = document.getElementById("opYearTo");
    var yearFrom = yearMin;
    var yearTo = yearMin;

    rngYearFrom.min = yearMin;
    rngYearFrom.max = yearMax;
    rngYearFrom.value = yearFrom;
    opYearFrom.value = yearFrom;

    rngYearTo.min = yearMin;
    rngYearTo.max = yearMax;
    rngYearTo.value = yearTo;
    opYearTo.value = yearTo;

  });
});


You see it! Both sliders are set to the minimum, 2016. And the output tags are showing.


We want these sliders to behave a certain way. They will both call the function setYear(), but with different arguments.
<label for="rngYearFrom">
  FROM
  <input id="rngYearFrom" type="range" oninput="setYear('from')" />
  <output id="opYearFrom" for="rngYearFrom"></output>
</label>
<br />
<label for="rngYearTo">
  TO
  <input id="rngYearTo" type="range" oninput="setYear('to')" />
  <output id="opYearTo" for="rngYearTo"></output>
</label>


In the JavaScript, we create this function. The idea here is that the value of rngDateFrom can never be greater than the value of rngDateTo. So rngDateTo's value needs to be adjusted to the value of rngdateFrom when that happens, and vice versa. if rngDateTo's value is less than that of rngDateFrom, the value of rngDateFrom needs to be adjusted to the value of rngdateTo.
    rngYearTo.min = yearMin;
    rngYearTo.max = yearMax;
    rngYearTo.value = yearTo;
    opYearTo.value = yearTo;
  });      
});

function setYear(target)
{

}


let currentData = [];
let yearSeriesData = [];


Let's begin by grabbing the required elements from the DOM - namely, the sliders and outputs.
function setYear(target)
{
  var rngYearFrom = document.getElementById("rngYearFrom");
  var rngYearTo = document.getElementById("rngYearTo");
  var opYearFrom = document.getElementById("opYearFrom");
  var opYearTo = document.getElementById("opYearTo");

}


And we grab the currently selected values.
function setYear(target)
{
  var rngYearFrom = document.getElementById("rngYearFrom");
  var rngYearTo = document.getElementById("rngYearTo");
  var opYearFrom = document.getElementById("opYearFrom");
  var opYearTo = document.getElementById("opYearTo");
  var yearFrom = parseInt(rngYearFrom.value);
  var yearTo = parseInt(rngYearTo.value);

}


Now, an If block handles the scenario of which slider was adjusted, based on the parameter, target. It's either "from" or "to".
function setYear(target)
{
  var rngYearFrom = document.getElementById("rngYearFrom");
  var rngYearTo = document.getElementById("rngYearTo");
  var opYearFrom = document.getElementById("opYearFrom");
  var opYearTo = document.getElementById("opYearTo");
  var yearFrom = parseInt(rngYearFrom.value);
  var yearTo = parseInt(rngYearTo.value);

  if (target == "from")
  {

  }
  else
  {
    
  }

}


Here, if the selected value of rngYearFrom, yearFrom, is greater than yearTo, that should not be allowed. We set yearTo to at least be equal to yearFrom. Then we adjust the slider and output. Don't forget to adjust the output for the "from" slider too.
function setYear(target)
{
  var rngYearFrom = document.getElementById("rngYearFrom");
  var rngYearTo = document.getElementById("rngYearTo");
  var opYearFrom = document.getElementById("opYearFrom");
  var opYearTo = document.getElementById("opYearTo");
  var yearFrom = parseInt(rngYearFrom.value);
  var yearTo = parseInt(rngYearTo.value);

  if (target == "from")
  {
    if (yearFrom > yearTo)
    {
      yearTo = yearFrom;
      rngYearTo.value = yearFrom;
      opYearTo.value = yearFrom;
    }

    opYearFrom.value = yearFrom;

  }
  else
  {
      
  }
}


And we do the reverse for the other slider!
function setYear(target)
{
  var rngYearFrom = document.getElementById("rngYearFrom");
  var rngYearTo = document.getElementById("rngYearTo");
  var opYearFrom = document.getElementById("opYearFrom");
  var opYearTo = document.getElementById("opYearTo");
  var yearFrom = parseInt(rngYearFrom.value);
  var yearTo = parseInt(rngYearTo.value);

  if (target == "from")
  {
    if (yearFrom > yearTo)
    {
      yearTo = yearFrom;
      rngYearTo.value = yearFrom;
      opYearTo.value = yearFrom;
    }

    opYearFrom.value = yearFrom;
  }
  else
  {
    if (yearTo < yearFrom)
    {
      yearFrom = yearTo;
      rngYearFrom.value = yearTo;
      opYearFrom.value = yearTo;
    }    

    opYearTo.value = yearTo;      

  }
}


Let's test this...

Both the Year FROM and TO start at 2016. Slide FROM to 2022. Does TO follow?


Now slide TO to a value lower than FROM, say, 2018. Does FROM follow?

Now slide TO to a value higher than FROM. FROM should stay put! In effect, FROM should always be lower than TO, or equal.


Rendering the Chart

Finally, eh? Create the renderLineChart() function.
  });      
});

function renderLineChart()
{

}


function setYear(target)
{


We get the values from the range sliders, and assign them to the variables yearFrom and yearTo, after coercing them to integers using the parseInt() function.
function renderLineChart()
{
  var rngYearFrom = document.getElementById("rngYearFrom");
  var rngYearTo = document.getElementById("rngYearTo");
  var yearFrom = parseInt(rngYearFrom.value);
  var yearTo = parseInt(rngYearTo.value);

}


We then declare dataset. It will be the subset of currentData whose year column (the first one at index 0) confirms to the range between yearFrom and yearTo, inclusive. We use the filter() method for this.
function renderLineChart()
{
  var rngYearFrom = document.getElementById("rngYearFrom");
  var rngYearTo = document.getElementById("rngYearTo");
  var yearFrom = parseInt(rngYearFrom.value);
  var yearTo = parseInt(rngYearTo.value);

  var dataset = currentData.filter((x) =>{ return parseInt(x[0]) >= yearFrom && parseInt(x[0]) <= yearTo});
}


Next, we declare years. This is actually a label that shows both the month and the year. For this, we use the second column (index 1) and the first column (index 0). We run the second column's value through the monthToName() function.
function renderLineChart()
{
  var rngYearFrom = document.getElementById("rngYearFrom");
  var rngYearTo = document.getElementById("rngYearTo");
  var yearFrom = parseInt(rngYearFrom.value);
  var yearTo = parseInt(rngYearTo.value);

  var dataset = currentData.filter((x) =>{ return parseInt(x[0]) >= yearFrom && parseInt(x[0]) <= yearTo});
  var years = dataset.map(col => monthToName(col[1]) + " " + col[0]);
}


Here's the function. It really isn't anything special - just returns a month string based on the integer passed into the function.
function setYear(target)
{
  var rngYearFrom = document.getElementById("rngYearFrom");
  var rngYearTo = document.getElementById("rngYearTo");
  var opYearFrom = document.getElementById("opYearFrom");
  var opYearTo = document.getElementById("opYearTo");
  var yearFrom = parseInt(rngYearFrom.value);
  var yearTo = parseInt(rngYearTo.value);

  if (target == "from")
  {
    if (yearFrom > yearTo)
    {
      yearTo = yearFrom;
      rngYearTo.value = yearFrom;
      opYearTo.value = yearFrom;
    }

    opYearFrom.value = yearFrom;
  }
  else
  {
    if (yearTo < yearFrom)
    {
      yearFrom = yearTo;
      rngYearFrom.value = yearTo;
      opYearFrom.value = yearTo;
    }    

    opYearTo.value = yearTo;      
  }
}

function monthToName(month)
{
  monthNames = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"];
  return monthNames[parseInt(month) - 1];
}


let currentData = [];
let yearSeriesData = [];


Now we have series, another array. For now, set series to contain one single object. Here are its properties.

name - this is the label that will appear on the chart's y-axis.
type - the line type. I chose "spline" because it's smooth and sexy.
data - the array of values. This is an empty array for now.
lineColor, lineWidth and dashStyle - Aesthetic choices. I went with a thick orange line.
marker - I don't want any damn markers, so it's an object with fillColor set to none.
function renderLineChart()
{
  var rngYearFrom = document.getElementById("rngYearFrom");
  var rngYearTo = document.getElementById("rngYearTo");
  var yearFrom = parseInt(rngYearFrom.value);
  var yearTo = parseInt(rngYearTo.value);

  var dataset = currentData.filter((x) =>{ return parseInt(x[0]) >= yearFrom && parseInt(x[0]) <= yearTo});
  var years = dataset.map(col => monthToName(col[1]) + " " + col[0]);

  var series = [];

  series = [
    {
      name: "commits",
      type: "spline",
      data:[],
      lineColor: "rgba(250, 100, 0, 1)",
      lineWidth: 5,
      dashStyle: "Solid",
      marker:
      {
        fillColor: "none"
      }
    }
  ];

}


Now, to render the chart! We will use the container div as the target here, for the Highchart object's chart() method.
function renderLineChart()
{
  var rngYearFrom = document.getElementById("rngYearFrom");
  var rngYearTo = document.getElementById("rngYearTo");
  var yearFrom = parseInt(rngYearFrom.value);
  var yearTo = parseInt(rngYearTo.value);

  var dataset = currentData.filter((x) =>{ return parseInt(x[0]) >= yearFrom && parseInt(x[0]) <= yearTo});
  var years = dataset.map(col => monthToName(col[1]) + " " + col[0]);

  var series = [];

  series = [
    {
      name: "commits",
      type: "spline",
      data: [],
      lineColor: "rgba(250, 100, 0, 1)",
      lineWidth: 5,
      dashStyle: "Solid",
      marker:
      {
        fillColor: "none"
      }
    }
  ];

  const chart = Highcharts.chart("container", {

  });

}


The first three properties we pass in, are chart, title and subtitle. As you can see, it's all visual styling and aesthetics. I've gone with an orange color scheme. (surprise, surprise)
function renderLineChart()
{
  var rngYearFrom = document.getElementById("rngYearFrom");
  var rngYearTo = document.getElementById("rngYearTo");
  var yearFrom = parseInt(rngYearFrom.value);
  var yearTo = parseInt(rngYearTo.value);

  var dataset = currentData.filter((x) =>{ return parseInt(x[0]) >= yearFrom && parseInt(x[0]) <= yearTo});
  var years = dataset.map(col => monthToName(col[1]) + " " + col[0]);
  var commits = dataset.map(col => col[2]);

  var series = [];

  series = [
    {
      name: "commits",
      type: "spline",
      data: commits,
      lineColor: "rgba(250, 100, 0, 1)",
      lineWidth: 5,
      dashStyle: "Solid",
      marker:
      {
        fillColor: "none"
      }
    }
  ];

  const chart = Highcharts.chart("container", {
    chart:
    {
      borderColor: "rgba(250, 100, 0, 1)",
      borderRadius: 10,
      borderWidth: 2,
    },
    title:
    {
      text: "My Contributions",
      style: { "color": "rgba(250, 100, 0, 1)", "font-size": "2.5em", "font-weight": "bold" }
    },
    subtitle:
    {
      text: "GitHub statistics by TeochewThunder",
      style: { "color": "rgba(250, 100, 0, 0.8)", "font-size": "0.8em" }
    }

  });
}


The next two properties define the labelling and scale. For xAxis, we pass in the years array as the value of the category property. For yAxis, it's all color scheme and labelling. Feel free to play with different values.
function renderLineChart()
{
  var rngYearFrom = document.getElementById("rngYearFrom");
  var rngYearTo = document.getElementById("rngYearTo");
  var yearFrom = parseInt(rngYearFrom.value);
  var yearTo = parseInt(rngYearTo.value);

  var dataset = currentData.filter((x) =>{ return parseInt(x[0]) >= yearFrom && parseInt(x[0]) <= yearTo});
  var years = dataset.map(col => monthToName(col[1]) + " " + col[0]);

  var series = [];

  series = [
    {
      name: "commits",
      type: "spline",
      data: [],
      lineColor: "rgba(250, 100, 0, 1)",
      lineWidth: 5,
      dashStyle: "Solid",
      marker:
      {
        fillColor: "none"
      }
    }
  ];

  const chart = Highcharts.chart("container", {
    chart:
    {
      borderColor: "rgba(250, 100, 0, 1)",
      borderRadius: 10,
      borderWidth: 2,
    },
    title:
    {
      text: "My Contributions",
      style: { "color": "rgba(250, 100, 0, 1)", "font-size": "2.5em", "font-weight": "bold" }
    },
    subtitle:
    {
      text: "GitHub statistics by TeochewThunder",
      style: { "color": "rgba(250, 100, 0, 0.8)", "font-size": "0.8em" }
    },
    xAxis:
    {
      categories: years
    },
    yAxis:
    {
      title:
      {
        text: "Commits"
      },
      gridLineColor: "rgba(250, 100, 0, 0.2)",
      tickColor: "rgba(250, 100, 0, 0.2)"
    }

  });
}


And finally, for the series property, we pass in series, which we created earlier.
function renderLineChart()
{
  var rngYearFrom = document.getElementById("rngYearFrom");
  var rngYearTo = document.getElementById("rngYearTo");
  var yearFrom = parseInt(rngYearFrom.value);
  var yearTo = parseInt(rngYearTo.value);

  var dataset = currentData.filter((x) =>{ return parseInt(x[0]) >= yearFrom && parseInt(x[0]) <= yearTo});
  var years = dataset.map(col => monthToName(col[1]) + " " + col[0]);

  var series = [];

  series = [
    {
      name: "commits",
      type: "spline",
      data: [],
      lineColor: "rgba(250, 100, 0, 1)",
      lineWidth: 5,
      dashStyle: "Solid",
      marker:
      {
        fillColor: "none"
      }
    }
  ];

  const chart = Highcharts.chart("container", {
    chart:
    {
      borderColor: "rgba(250, 100, 0, 1)",
      borderRadius: 10,
      borderWidth: 2,
    },
    title:
    {
      text: "My Contributions",
      style: { "color": "rgba(250, 100, 0, 1)", "font-size": "2.5em", "font-weight": "bold" }
    },
    subtitle:
    {
      text: "GitHub statistics by TeochewThunder",
      style: { "color": "rgba(250, 100, 0, 0.8)", "font-size": "0.8em" }
    },
    xAxis:
    {
      categories: years
    },
    yAxis:
    {
      title:
      {
        text: "Commits"
      },
      gridLineColor: "rgba(250, 100, 0, 0.2)",
      tickColor: "rgba(250, 100, 0, 0.2)"

    },
    series: series
  });
}


Call the function here...
document.addEventListener("DOMContentLoaded", function () {
  fetch("http://www.teochewthunder.com/demo/hc_github/hcdata_github.csv")
  .then(response => response.text())
  .then(csvData => {
    const rows = csvData.split("\n");
    currentData = rows.slice(1).map(row => {
      const cols = row.split(",");
      var year = parseInt(cols[0]);
      var month = parseInt(cols[1]);
      var contributions = parseInt(cols[2]);

      return [year, month, contributions];
    });

    var years = currentData.map(col => col[0]);
    var yearMin = Math.min(...years);
    var yearMax = Math.max(...years);

    var yearFrom = yearMin;
    var yearTo = yearMin;

    var rngYearFrom = document.getElementById("rngYearFrom");
    var rngYearTo = document.getElementById("rngYearTo");
    var opYearFrom = document.getElementById("opYearFrom");
    var opYearTo = document.getElementById("opYearTo");

    rngYearFrom.min = yearMin;
    rngYearFrom.max = yearMax;
    rngYearFrom.value = yearFrom;
    opYearFrom.value = yearFrom;

    rngYearTo.min = yearMin;
    rngYearTo.max = yearMax;
    rngYearTo.value = yearTo;
    opYearTo.value = yearTo;

    renderLineChart();  
  });      
});


...and here, at the end of the setYear() function.
function setYear(target)
{
  var rngYearFrom = document.getElementById("rngYearFrom");
  var rngYearTo = document.getElementById("rngYearTo");
  var opYearFrom = document.getElementById("opYearFrom");
  var opYearTo = document.getElementById("opYearTo");
  var yearFrom = parseInt(rngYearFrom.value);
  var yearTo = parseInt(rngYearTo.value);

  if (target == "from")
  {
    if (yearFrom > yearTo)
    {
      yearTo = yearFrom;
      rngYearTo.value = yearFrom;
      opYearTo.value = yearFrom;
    }

    opYearFrom.value = yearFrom;
  }
  else
  {
    if (yearTo < yearFrom)
    {
      yearFrom = yearTo;
      rngYearFrom.value = yearTo;
      opYearFrom.value = yearTo;
    }    

    opYearTo.value = yearTo;      
  }

  renderLineChart();
}


Remove the red lines.
div { outline:0px solid rgb(255, 0, 0); }


And here is the current placeholder we have for the chart. You should see right now that both range sliders are at the minimum value, 2016.



Next

Displaying chart data.

Wednesday, 12 November 2025

Spot The Bug: The Underperforming Chart Scale

Hey, hey! Spot The Bug is back in town, and we're rolling out a new bug-fix story!

Can't wait to
squash another bug.

Working with D3 is one of my favorite pasttimes, especially when I have a new dataset to play with. However, there are some gotchas, and today's episode of Spot The Bug is a story of how one of these gotchas, well, got me.

I had some data that I wanted to plot a chart for. I truncated it a bit for brevity, but it will serve to illustrate the point. Note in particular, the column val.

datavals.csv
date,val
2024-11-01,8
2024-11-02,2
2024-11-03,17
2024-11-04,11
2024-11-05,25
2024-11-06,4
2024-11-07,7
2024-11-08,3
2024-11-09,8
2024-11-10,15


Here's the code I wrote to create an y-axis
<!DOCTYPE html>
<html>
  <head>
    <title>Chart</title>

    <style>
      svg
      {
        width: 800px;
        height: 500px;
      }
      
      .scaleTick, .scale
      {
        stroke: rgba(100, 100, 100, 1);
        stroke-width: 1px;
      }
      
      .scaleText
      {
        font: 8px verdana;
        fill: rgba(100, 100, 100, 1);
        text-anchor: end;
      }
    </style>

    <script src="https://d3js.org/d3.v4.min.js"></script>
  </head>

  <body>
    <svg>

    </svg>

    <script>
      d3.csv("datavals.csv", function(data)
      {
        var dataVals = [];

       for (var i = 0; i < data.length; i++)
       {
         dataVals.push(data[i].val);
       }

       var maxVal = d3.max(dataVals);

        var chart = d3.select("svg");
        chart.html("");

        chart
     .append("line")
     .attr("class", "scale")
     .attr("x1", "50px")
     .attr("y1", "50px")
     .attr("x2", "50px")
     .attr("y2", "450px");
    
        var pxPerUnit = Math.floor(400 / maxVal);
    
        var scale = [];
        for (var i = 450; i >= 50; i -= pxPerUnit)
        {
          scale.push(i);
        }
    
     chart.selectAll("line.scaleTick")
     .data(scale)
     .enter()
     .append("line")
     .attr("class", "scaleTick")
     .attr("x1", "40px")
     .attr("y1", function(d)
     {
     return d + "px";
     })
     .attr("x2", "50px")
     .attr("y2", function(d)
     {
     return d + "px";
     });
    
     chart.selectAll("text.scaleText")
     .data(scale)
     .enter()
     .append("text")
     .attr("class", "scaleText")
     .attr("x", "30px")
     .attr("y", function(d)
     {
     return d + "px";
     })
     .text(
     function(d, i)
     {
     return i;
     });     
      });
    </script>
  </body>
</html>

Here, I got a scale. However, something was wrong. The largest value in my dataset was 25. Why was the largest value here 8?


What Went Wrong

My first clue was here, near the top of the script where I declared maxVal. Because according to this line, max() was returning 8 as the largest value.
var maxVal = d3.max(dataVals);


This would explain why the scale started with the value 8 and worked its way down from there.

Why It Went Wrong

If the largest value in the dataVals array was 25 and the script thought the largest value was 8, there was one explanation for that. The script was comparing alphabetically rather than numerically. As text, "8" is always larger than "25". Basically, the dataVals array was being filled with strings rather than integers from the CSV file.
var dataVals = [];

for (var i = 0; i < data.length; i++)
{
  dataVals.push(data[i].val);
}

How I Fixed It

I used the parseInt() function to convert the data before pushing it into the dataVals array.
var dataVals = [];

for (var i = 0; i < data.length; i++)
{
  dataVals.push(parseInt(data[i].val));
}


And now it took 25 as the largest value! Because now dataVals was filled with integers rather than text, and it was comparing accordingly.


Moral of the Story

You'd think I would have learned to sanitize these things by now, especially in JavaScript, or any loosely-typed language. Apparently not!

Stay int-tellectual,
T___T

Tuesday, 23 September 2025

Web Tutorial: NodeJS Text Replacement Blogging Tool

Writing content for the web can be tricky and tedious, because it involves converting text to HTML. And when delivering a web tutorial (such as the one I'm doing now) this increases tenfold due to special characters which could be mistaken for genuine HTML. Because web tutorials for the web frequently involve HTML, amirite? At first I was OK with doing text replacements on Sublime Text, but even with programmable macros and such, it rapidly became a repetitive chore.

So when I was exploring NodeJS, I came up with this absolutely genius idea. How about I create an interface to process my text and spit it out in blog-friendly format? I also needed this thing to be configurable in case my requirements evolved. Nothing I couldn't achieve with vanilla JavaScript. Except I didn't want to be making code changes every time my requirements changed. No, I needed the replacements to be read from a CSV file which I could change any given time.

Plus, doing it this way gives me the opportunity to introduce the core module fs and the installed module csv-parser.

Thus, I started my new blogging tool project. For this, I ran the following commands.
npm install --save express
npm install --save express-handlebars
npm install --save csv-parser


This is the code that includes Express as middleware and the setup for the port...

app.js
var express = require("express");

var app = express();

app.set("port", process.env.PORT || 3000);


...and the code that uses Handlebars as a templating engine.

app.js
var express = require("express");

var app = express();

var handlebars = require("express-handlebars").create({defaultLayout: "main"});
app.engine("handlebars", handlebars.engine);

app.set("view engine", "handlebars");

app.set("port", process.env.PORT || 3000);


Here, we ensure that form bodies can be parsed using Express to parse JSON. And also, we tell Express to use the assets directory for static links.

app.js
var express = require("express");

var app = express();

var handlebars = require("express-handlebars").create({defaultLayout: "main"});
app.engine("handlebars", handlebars.engine);

app.set("view engine", "handlebars");
app.set("port", process.env.PORT || 3000);

app.use(express.json());
app.use(express.urlencoded({ extended: true }));

app.use(express.static("assets"));


We then implement routes to handle 404s and general errors...

app.js
var express = require("express");

var app = express();

var handlebars = require("express-handlebars").create({defaultLayout: "main"});
app.engine("handlebars", handlebars.engine);

app.set("view engine", "handlebars");
app.set("port", process.env.PORT || 3000);

app.use(express.json());
app.use(express.urlencoded({ extended: true }));

app.use(express.static("assets"));

app.use((req, res, next)=> {
  res.status(404);
  res.render("404");
});

app.use((err, req, res, next)=> {
  res.status(500);
  res.render("500", { errorMessage: err.code });
});


Here's the route for form processing. We'll call it process and set it to POST. Leave empty for now.

app.js
var express = require("express");

var app = express();

var handlebars = require("express-handlebars").create({defaultLayout: "main"});
app.engine("handlebars", handlebars.engine);

app.set("view engine", "handlebars");
app.set("port", process.env.PORT || 3000);

app.use(express.json());
app.use(express.urlencoded({ extended: true }));

app.use(express.static("assets"));

app.post("/process", async (req, res)=> {

});

app.use((req, res, next)=> {
  res.status(404);
  res.render("404");
});

app.use((err, req, res, next)=> {
  res.status(500);
  res.render("500", { errorMessage: err.code });
});


And finally, for routes, we have home, a GET route. Inside it, we will render the form view with some data.

app.js
var express = require("express");

var app = express();

var handlebars = require("express-handlebars").create({defaultLayout: "main"});
app.engine("handlebars", handlebars.engine);

app.set("view engine", "handlebars");
app.set("port", process.env.PORT || 3000);

app.use(express.json());
app.use(express.urlencoded({ extended: true }));

app.use(express.static("assets"));

app.get("/", (req, res)=> {
  res.render("form", { textContent: "", btnCLass: "", message: "Paste your text in the box provided, then hit the PROCESS button." });
});


app.post("/process", async (req, res)=> {

});

app.use((req, res, next)=> {
  res.status(404);
  res.render("404");
});

app.use((err, req, res, next)=> {
  res.status(500);
  res.render("500", { errorMessage: err.code });
});



These are the files I have for rendering pages, in the views directory. Firstly, the layout file main.handlebars, which we specified in app.js.

views/layout/main.handlebars
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>T___T's Text Replacement Tool for Blogging</title>

    <link rel="stylesheet" type="text/css" href="css/styles.css">
  </head>
  <body>
    <h1>TEXT REPLACE TOOL</h1>
    <div class="content">
      {{{ body }}}  
    </div>    
  </body>
</html>


The rest are pretty standard. The 404 view is next.

views/404.handlebars
<h1>404</h1>

<p>Not found!</p>


And for 500.

views/500.handlebars
<h1>500</h1>

<p>There was an error.</p>
<p><b>{{ errorMessage }}</b></p>


And this! The form view. We have a form that submits a POST to the process route.

views/form.handlebars
<form action="/process" method="POST">

</form>


Then a textarea tag with names and id txtTextToProcess. In it, we will display the data textContent.

views/form.handlebars
<form action="/process" method="POST">
  <textarea id="txtTextToProcess" name="txtTextToProcess" required>{{ textContent }}</textarea>
</form>


We then follow up by displaying the data message.

views/form.handlebars
<form action="/process" method="POST">
  <textarea id="txtTextToProcess" name="txtTextToProcess" required>{{ textContent }}</textarea>

  <br />
  {{ message }}
  <br />

</form>


And finally the SUBMIT button, which will be styled using the CSS class btnClass.

views/form.handlebars
<form action="/process" method="POST">
  <textarea id="txtTextToProcess" name="txtTextToProcess" required>{{ textContent }}</textarea>

  <br />
  {{ message }}
  <br />
  <button class="{{ btnClass }}">PROCESS</button>
</form>


This is the CSS file for the app, and honestly there's not much here because it's going to be substance over style. Meaning, ugly. It's in the assets directory, which we earlier specified in app.js that Express should use for remote file linking. The really important thing here is hidden, which will hide whatever it's applied to. The rest is just... fluff. And not even particularly pretty fluff.

assets/css/styles.css
.content
{
  width: 90%;
  height: 500px;
  margin: 10px auto 0 auto;
}

textarea
{
  width: 90%;
  height: 300px;
  margin: 10px auto 0 auto;
}

button
{
  width: 10em;
  height: 1.5em;
  display: inline-block;
  float: right;
}

.hidden
{
  display: none;
}


There, you should be able to see this, at least, when you run "node app.js" in the CLI.


This is the CSV file I'm using. You'll see all the replacements. The first few are straightforward enough - "<" being replaced by "lt" and ">" being replaced by "gt".

assets/csv/inputs.csv
find,replace
"<","<"
">",">"


The next couple are a bit more advanced. We want to replace tabs with two HTML spaces. And new lines with HTML break. In these cases, we have to escape the special characters.

assets/csv/inputs.csv
find,replace
"<","<"
">",">"
"\t","  "
"\r\n","<br />"


Here, I specify some shorthand that I use in my blogging. When I have, for example, "c---" followed by a br tag (because after carrying out the previous replacements, all new lines would be HTML breaks) I want it to be replaced with a div tag styled using the CSS classes post_box and code. Note that the class attribute value here would be encased in double quotes... and each literal double quote has to be escaped using another double quote.

assets/csv/inputs.csv
find,replace
"<","<"
">",">"
"\t","  "
"\r\n","<br />"
"c---<br />","<div class=""post_box code"">"
"r---<br />","<div class=""post_box result"">"
"i---<br />","<div class=""post_box info"">"
"s---<br />","<div class=""signature"">"


Lastly, all occurences of "e---" and a br tag need to be replaced by a closing div tag.

assets/csv/inputs.csv
find,replace
"<","<"
">",">"
"\t","  "
"\r\n","<br />"
"c---<br />","<div class=""post_box code"">"
"r---<br />","<div class=""post_box result"">"
"i---<br />","<div class=""post_box info"">"
"s---<br />","<div class=""signature"">"
"<br />e---","</div>"


Now we start to prepare the code for processing data, in the POST route. We declare processedText, and set it to the value of the textarea that was sent in the POST, txtTextToProcess.

app.js
app.post("/process", async (req, res)=> {
  let processedText = req.body.txtTextToProcess;
});


At the end of this, you want to render form but with processedText as your textContent. You want the button to be invisible, so set btnClass to hidden, and the message property should be just a string indicating success.

app.js
app.post("/process", async (req, res)=> {
  let processedText = req.body.txtTextToProcess;

  res.render("form", { textContent: processedText, btnClass: "hidden", message: "Text processed." });
});


But of course, we will be working on processedText. For this, we call the asynchronous function loadChanges(), using await to pause execution until it's done running.

app.js
app.post("/process", async (req, res)=> {
  let processedText = req.body.txtTextToProcess;
  await loadChanges();

  res.render("form", { textContent: processedText, btnClass: "hidden", message: "Text processed." });
});


Here, we declare the global array changes. Then we create the asynchronous function loadChanges().

app.js
const fs = require("fs");
const csv = require("csv-parser");

let changes = [];

async function loadChanges() {

}


app.get("/", (req, res)=> {
  res.render("form", { textContent: "", btnCLass: "", message: "Paste your text in the box provided, then hit the PROCESS button." });
});

app.post("/process", async (req, res)=> {
  let processedText = req.body.txtTextToProcess;
  await loadChanges();

  res.render("form", { textContent: processedText, btnClass: "hidden", message: "Text processed." });
});


Here, we have a Try-catch block. We'll try reading the CSV file, and then do some logging if it fails.

app.js
let changes = [];

async function loadChanges() {
  try {

  } catch (err) {
    throw new Error("Error reading CSV.");
    console.error("Error reading CSV:", err);
    }
}


The main action here is to run the asynchronous function loadFile(), which we will create, and pass in the file path as an argument. The returned value should be assigned to the changes array.

app.js
let changes = [];

async function loadChanges() {
  try {
    changes = await loadFile("assets/csv/inputs.csv");
  } catch (err) {
      throw new Error("Error reading CSV.");
      console.error("Error reading CSV:", err);
    }
}


loadFile() is another async function. It has a parameter, filePath. It returns a Promise object.

app.js
let changes = [];

async function loadFile(filePath) {
  return new Promise((resolve, reject) => {

  });
}


async function loadChanges() {
  try {
    changes = await loadFile("assets/csv/inputs.csv");
  } catch (err) {
    throw new Error("Error reading CSV.");
    console.error("Error reading CSV:", err);
  }
}

We will first declare the array results.

app.js
async function loadFile(filePath) {
  return new Promise((resolve, reject) => {
    const results = [];

  });
}


We then call the createReadStream() method of fs, passing in filePath as an argument. The result will be run through the pipe() method, which connects the resultant stream of data to something else.

app.js
async function loadFile(filePath) {
  return new Promise((resolve, reject) => {
    const results = [];

    fs.createReadStream(filePath)
    .pipe()
  });
}


In this case, the connection is to csv(). csv() is a CSV parser, simply put, and running pipe() with csv() as an argument means that we're reading the file stream as a CSV.

app.js
async function loadFile(filePath) {
  return new Promise((resolve, reject) => {
    const results = [];

    fs.createReadStream(filePath)
    .pipe(csv())
  });
}


Now we have a callback for each row that fs is processing. We basically push row into the results array.

app.js
async function loadFile(filePath) {
  return new Promise((resolve, reject) => {
    const results = [];

    fs.createReadStream(filePath)
    .pipe(csv())
    .on("data", (row) => {
      results.push(row);
    })
  });
}


But before that, since some of the characters in the find column need to be unescaped, we run the values through the unescapeSpecialChars() function.

app.js
async function loadFile(filePath) {
  return new Promise((resolve, reject) => {
    const results = [];

    fs.createReadStream(filePath)
    .pipe(csv())
    .on("data", (row) => {
      row.find = unescapeSpecialChars(row.find);
      results.push(row);
    })
  });
}


Then we handle errors and resolutions.

app.js
async function loadFile(filePath) {
  return new Promise((resolve, reject) => {
    const results = [];

    fs.createReadStream(filePath)
    .pipe(csv())
    .on("data", (row) => {
      row.find = unescapeSpecialChars(row.find);
      results.push(row);
    })
    .on("end", () => resolve(results))
    .on("error", (err) => reject(err));

  });
}


This is the unescapeSpecialChars() function. Nothing special here. We just accept a string parameter and return the result after replacing the characters we're looking for, with their unescaped equivalents. We need to do this because the characters were formatted a certain way to fit into the CSV.

app.js
async function loadChanges() {
  try {
    changes = await loadFile("assets/csv/inputs.csv");
  } catch (err) {
    throw new Error("Error reading CSV.");
    console.error("Error reading CSV:", err);
  }
}

function unescapeSpecialChars(str) {
   return str
  .replace(/\\t/g, "\t")
  .replace(/\\r\\n/g, "\r\n")
  .replace(/\\n/g, "\n")
  .replace(/\\r/g, "\r");
}  

app.get("/", (req, res)=> {
  res.render("form", { textContent: "", btnCLass: "", message: "Paste your text in the box provided, then hit the PROCESS button." });
});


Let's test this! Add some HTML in the textbox.


Click the PROCESS button, and you'll see the "<" and ">" symbols have been replaced.


Now let's test new lines and tabs.


See the new lines replaced by br tags and tabs replaced by HTML spaces.


For our final trick, we add this shorthand.

And we can see that this has been replaced by the appropraite opening and closing div tags!



That's it...

And of course, from this point on, all I need to do is copy the text and paste it as HTML.

This little beauty has been inestimably useful. I can't even begin to imagine blog maintenance without it now.


Good luck out there. <br /> a leg!
T___T